[GH-ISSUE #6799] [Bug] Hidden ARM64 slices from Apple lipo trigger a false FAT-table integrity failure #4147

New issue

Closed

opened 2026-10-31 00:47:10 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:47:10 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6799

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The production MachOSliceArtifactReader rejects an Apple lipo -hideARM64 container layout with ArtifactReaderFailure(reason: "integrity", message: "lipo architecture count disagrees with the Mach-O FAT table"). Apple intentionally excludes the hidden ARM64 record from the header's declared count, while its -detailed_info output includes that record.

This was executed at the production reader boundary on Linux with simulated Apple lipo output and an inert, header-only FAT fixture. The native macOS CLI and Apple lipo process were not run. The real-format compatibility conclusion is source-derived from released Apple cctools; this does not establish prevalence in shipped applications.

Reproduction

With the checkout's locked dependencies installed, run from its root:

node --import tsx --input-type=module <<'JS'
import { inflateSync } from 'node:zlib';
import { mkdtemp, writeFile, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { MachOSliceArtifactReader } from './src/artifacts/MachOSliceArtifactReader.ts';
const root = await mkdtemp(join(tmpdir(), 'rea-hidden-arm64-'));
const fixture = Buffer.from('eJzt2bEJgEAMBdBoJ1g4gIX7j2DhABY6iQtYiCd3drYiwnuQC0l17SfzMU4RUaVqUzURXWrRX3OVd8lwP2UGAAAA/mTZt6ME/6i//gwAAADwijXn/+v+/5j/T/SBC3U=', 'base64');
const hidden = inflateSync(fixture);
try {
  for (const hide of [false, true]) {
    const bytes = Buffer.from(hidden); bytes.writeUInt32BE(hide ? 1 : 2, 4);
    const path = join(root, hide ? 'hidden' : 'ordinary'); await writeFile(path, bytes);
    const stdout = `fat_magic 0xcafebabe\nnfat_arch ${hide ? '1 (+1 hidden)' : '2'}\narchitecture armv7\n cputype CPU_TYPE_ARM\n cpusubtype CPU_SUBTYPE_ARM_V7\n offset 4096\n size 28\n align 2^12 (4096)\narchitecture arm64${hide ? ' (hidden)' : ''}\n cputype CPU_TYPE_ARM64\n cpusubtype CPU_SUBTYPE_ARM64_ALL\n offset 8192\n size 32\n align 2^12 (4096)\n`;
    const reader = new MachOSliceArtifactReader(path, {}, { run: async () => ({ok:true,value:{tool:'lipo',executable:'/usr/bin/lipo',executableSha256:'1'.repeat(64),toolVersion:null,versionReason:'simulated Apple output',arguments:[],stdout,stderr:'',stdoutBytes:Buffer.byteLength(stdout),stderrBytes:0,exitCode:0,signal:null}}) });
    try { const entries=[]; for await (const e of reader.entries()) entries.push(e.path); console.log(hide ? 'hidden' : 'ordinary', entries); }
    catch (e) { console.log(hide ? 'hidden' : 'ordinary', e.reason, e.message); }
    finally { await reader.close(); }
  }
} finally { await rm(root, {recursive:true,force:true}); }
JS

The 8,224-byte fixture contains ARMv7 and ARM64 MH_EXECUTE headers at offsets 4096 and 8192, with sizes 28 and 32 and alignment 4096. It is never launched. The two inputs differ only at byte 7, changing nfat_arch between 2 and 1. The simulated output follows Apple's released -detailed_info formatting, including nfat_arch 1 (+1 hidden) and architecture arm64 (hidden).

One comprehensive reader-boundary probe and the reduced ordinary/hidden reproduction above were executed before this draft. Additional controls used the same hidden bytes with only the visible lipo row, and an ordinary two-slice file with an incorrect reported size.

Expected behavior

Inventory the valid hidden ARM64 layout without calling its intentional slice-count convention an integrity contradiction. Preserve each observed slice's actual range and distinguish unsupported compatibility from corrupted bytes if support is unavailable.

Apple cctools explicitly writes this layout, recovers the hidden record, and prints it in detailed output. These references are from published tag cctools-1035.1.102, not an unreleased branch. REA's static-inspection contract promises universal-slice inventory and extraction with physical identity checks.

Impact and scope

The observed reader path refuses the entire inventory rather than retaining either slice. The same reader supplies native macOS universal-Mach-O inventory and extraction. Compatibility with actual Apple-generated hidden binaries is source-derived; no real macOS workflow, application corpus, or occurrence rate was tested. A normal two-visible-slice representation passes. Recreating a copy without -hideARM64 is a possible workaround, but that Apple command was not tested.

REA version

6.7.0; source revision bd4f5e9c197ba942ed026c9fed6ff3fae663901d.

Runtime and platform

Node.js v26.12.1; Linux x64; locked tsx 4.23.15. Simulated Apple lipo runner; no native Apple tool process or target execution.

Target and artifact context

Self-created inert FAT Mach-O fixture modeling Apple's backward-compatible hidden ARM64 entry convention. ARMv7 subtype 9 and ordinary ARM64 subtype 0; no arm64e ABI variant.

Interface

Other

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

ordinary [ 'slices/armv7', 'slices/arm64' ]
hidden integrity lipo architecture count disagrees with the Mach-O FAT table

The comprehensive probe also observed:

ordinary: structural slices=2, producer slices=2, two entries returned
hidden: structural slices=1, producer slices=2, integrity failure
same hidden bytes / visible producer row only: one armv7 entry returned
ordinary / incorrect reported armv7 size: integrity failure for that size mismatch

The structural reader follows the raw declared FAT count. The artifact reader then requires that count to equal all parsed lipo rows, before comparing individual ranges.

Additional context

Distinct from #4280: that report concerns arm64e.x1 subtype naming; this fixture uses ordinary arm64 subtype 0 and fails at the earlier slice-count comparison. #5191 concerns format/platform vocabulary and collapsed architecture names, rather than Apple's hidden-entry count convention. No open inventory title/body matching hideARM64, raw_nfat, or hidden architecture/slice handling identified a matching report.

Triage assessment

Impact level: P2
Assessment status: provisional
Impact basis: The production reader demonstrably rejects this documented Apple container convention with a false integrity failure; actual native macOS reproduction and real-application frequency remain unverified.
Workaround status: unverified
Workaround basis: The equivalent two-visible-slice fixture succeeds, suggesting that recreating a copy without hidden entries would work; no real Apple lipo normalization command was executed.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6799 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The production `MachOSliceArtifactReader` rejects an Apple `lipo -hideARM64` container layout with `ArtifactReaderFailure(reason: "integrity", message: "lipo architecture count disagrees with the Mach-O FAT table")`. Apple intentionally excludes the hidden ARM64 record from the header's declared count, while its `-detailed_info` output includes that record. This was executed at the production reader boundary on Linux with simulated Apple lipo output and an inert, header-only FAT fixture. The native macOS CLI and Apple lipo process were not run. The real-format compatibility conclusion is source-derived from released Apple cctools; this does not establish prevalence in shipped applications. ## Reproduction With the checkout's locked dependencies installed, run from its root: ```sh node --import tsx --input-type=module <<'JS' import { inflateSync } from 'node:zlib'; import { mkdtemp, writeFile, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { MachOSliceArtifactReader } from './src/artifacts/MachOSliceArtifactReader.ts'; const root = await mkdtemp(join(tmpdir(), 'rea-hidden-arm64-')); const fixture = Buffer.from('eJzt2bEJgEAMBdBoJ1g4gIX7j2DhABY6iQtYiCd3drYiwnuQC0l17SfzMU4RUaVqUzURXWrRX3OVd8lwP2UGAAAA/mTZt6ME/6i//gwAAADwijXn/+v+/5j/T/SBC3U=', 'base64'); const hidden = inflateSync(fixture); try { for (const hide of [false, true]) { const bytes = Buffer.from(hidden); bytes.writeUInt32BE(hide ? 1 : 2, 4); const path = join(root, hide ? 'hidden' : 'ordinary'); await writeFile(path, bytes); const stdout = `fat_magic 0xcafebabe\nnfat_arch ${hide ? '1 (+1 hidden)' : '2'}\narchitecture armv7\n cputype CPU_TYPE_ARM\n cpusubtype CPU_SUBTYPE_ARM_V7\n offset 4096\n size 28\n align 2^12 (4096)\narchitecture arm64${hide ? ' (hidden)' : ''}\n cputype CPU_TYPE_ARM64\n cpusubtype CPU_SUBTYPE_ARM64_ALL\n offset 8192\n size 32\n align 2^12 (4096)\n`; const reader = new MachOSliceArtifactReader(path, {}, { run: async () => ({ok:true,value:{tool:'lipo',executable:'/usr/bin/lipo',executableSha256:'1'.repeat(64),toolVersion:null,versionReason:'simulated Apple output',arguments:[],stdout,stderr:'',stdoutBytes:Buffer.byteLength(stdout),stderrBytes:0,exitCode:0,signal:null}}) }); try { const entries=[]; for await (const e of reader.entries()) entries.push(e.path); console.log(hide ? 'hidden' : 'ordinary', entries); } catch (e) { console.log(hide ? 'hidden' : 'ordinary', e.reason, e.message); } finally { await reader.close(); } } } finally { await rm(root, {recursive:true,force:true}); } JS ``` The 8,224-byte fixture contains ARMv7 and ARM64 `MH_EXECUTE` headers at offsets 4096 and 8192, with sizes 28 and 32 and alignment 4096. It is never launched. The two inputs differ only at byte 7, changing `nfat_arch` between 2 and 1. The simulated output follows Apple's released `-detailed_info` formatting, including `nfat_arch 1 (+1 hidden)` and `architecture arm64 (hidden)`. One comprehensive reader-boundary probe and the reduced ordinary/hidden reproduction above were executed before this draft. Additional controls used the same hidden bytes with only the visible lipo row, and an ordinary two-slice file with an incorrect reported size. ## Expected behavior Inventory the valid hidden ARM64 layout without calling its intentional slice-count convention an integrity contradiction. Preserve each observed slice's actual range and distinguish unsupported compatibility from corrupted bytes if support is unavailable. Apple cctools explicitly [writes this layout](https://github.com/apple-oss-distributions/cctools/blob/e0d56624eca2a76c2ace4c21850df9e666de4ca5/misc/lipo.c#L986-L1015), [recovers the hidden record](https://github.com/apple-oss-distributions/cctools/blob/e0d56624eca2a76c2ace4c21850df9e666de4ca5/misc/lipo.c#L1223-L1249), and [prints it in detailed output](https://github.com/apple-oss-distributions/cctools/blob/e0d56624eca2a76c2ace4c21850df9e666de4ca5/misc/lipo.c#L833-L880). These references are from published tag `cctools-1035.1.102`, not an unreleased branch. REA's [static-inspection contract](https://github.com/morluto/rea/blob/bd4f5e9c197ba942ed026c9fed6ff3fae663901d/docs/native-investigation.md#L345-L355) promises universal-slice inventory and extraction with physical identity checks. ## Impact and scope The observed reader path refuses the entire inventory rather than retaining either slice. The same reader supplies native macOS universal-Mach-O inventory and extraction. Compatibility with actual Apple-generated hidden binaries is source-derived; no real macOS workflow, application corpus, or occurrence rate was tested. A normal two-visible-slice representation passes. Recreating a copy without `-hideARM64` is a possible workaround, but that Apple command was not tested. ## REA version 6.7.0; source revision `bd4f5e9c197ba942ed026c9fed6ff3fae663901d`. ## Runtime and platform Node.js v26.12.1; Linux x64; locked tsx 4.23.15. Simulated Apple lipo runner; no native Apple tool process or target execution. ## Target and artifact context Self-created inert FAT Mach-O fixture modeling Apple's backward-compatible hidden ARM64 entry convention. ARMv7 subtype 9 and ordinary ARM64 subtype 0; no arm64e ABI variant. ## Interface Other ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence ```text ordinary [ 'slices/armv7', 'slices/arm64' ] hidden integrity lipo architecture count disagrees with the Mach-O FAT table ``` The comprehensive probe also observed: ```text ordinary: structural slices=2, producer slices=2, two entries returned hidden: structural slices=1, producer slices=2, integrity failure same hidden bytes / visible producer row only: one armv7 entry returned ordinary / incorrect reported armv7 size: integrity failure for that size mismatch ``` The [structural reader](https://github.com/morluto/rea/blob/bd4f5e9c197ba942ed026c9fed6ff3fae663901d/src/artifacts/apple/MachoLoadCommandReader.ts#L169-L188) follows the raw declared FAT count. [The artifact reader](https://github.com/morluto/rea/blob/bd4f5e9c197ba942ed026c9fed6ff3fae663901d/src/artifacts/MachOSliceArtifactReader.ts#L138-L145) then requires that count to equal all parsed lipo rows, before comparing individual ranges. ## Additional context Distinct from #4280: that report concerns `arm64e.x1` subtype naming; this fixture uses ordinary `arm64` subtype 0 and fails at the earlier slice-count comparison. #5191 concerns format/platform vocabulary and collapsed architecture names, rather than Apple's hidden-entry count convention. No open inventory title/body matching `hideARM64`, `raw_nfat`, or hidden architecture/slice handling identified a matching report. ## Triage assessment - Impact level: P2 - Assessment status: provisional - Impact basis: The production reader demonstrably rejects this documented Apple container convention with a false integrity failure; actual native macOS reproduction and real-application frequency remain unverified. - Workaround status: unverified - Workaround basis: The equivalent two-visible-slice fixture succeeds, suggesting that recreating a copy without hidden entries would work; no real Apple lipo normalization command was executed.

bsakaguchi closed this issue

2026-10-31 00:47:11 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:45 +00:00

[PR #4147] [CLOSED] fix(mcp): advertise tool inputs without shared $defs #6087