morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6741] [Bug] Archive content nodes discard observed distant FAT slice headers #4103

New issue

Closed

opened 2026-10-31 00:46:50 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:50 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6741

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The shared artifact inventory workflow returns conflicting format facts for an ordinary ZIP member containing a synthetic FAT Mach-O image. The member occurrence is mach-o-universal with a verified hash, but its linked content node is unknown. Inspecting the identical bytes as the root produces mach-o-universal for the same content artifact ID. A Godot PCK v3 member produces the same mismatch. Both archive inventories report complete container-entry coverage.

This was observed when the thin arm64 slice started at offset 16,384, beyond the scanner's 8,192-byte classification prefix. A 4,096-offset control produces matching mach-o-universal formats throughout.

Reproduction

From the source revision below with its locked dependencies installed, save the following as $WORK/probe-fat-node.mjs, where $WORK is an existing disposable directory outside the checkout. Run:

node --import ./node_modules/tsx/dist/loader.mjs "$WORK/probe-fat-node.mjs" "$WORK/evidence"

This is the probe actually run, once for each near/far ZIP/PCK case and once per direct-root control. It only writes and inventories inert authored bytes.

import assert from 'node:assert/strict';
import { mkdir, writeFile } from 'node:fs/promises';
import { join, resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
import { createRequire } from 'node:module';
import { execFileSync } from 'node:child_process';

const root = resolve(process.env.REA_SOURCE ?? process.cwd());
const evidenceDir = resolve(process.argv[2] ?? '.');
await mkdir(evidenceDir, { recursive: true });
const source = (path) => import(pathToFileURL(join(root, path)).href);
const { inventoryArtifact } = await source('src/artifacts/inventory/ArtifactInventory.ts');
const { ArtifactResourceScope } = await source('src/artifacts/ArtifactResourceScope.ts');
const { buildGodotPck } = await source('tests/fixtures/godotPck.ts');
const require = createRequire(join(root, 'package.json'));
const { ZipWriter, Uint8ArrayWriter, Uint8ArrayReader } = await import(require.resolve('@zip.js/zip.js'));

function fat(offset) {
  const data = Buffer.alloc(offset + 48);
  data.writeUInt32BE(0xcafebabe, 0);
  data.writeUInt32BE(1, 4);
  data.writeUInt32BE(0x0100000c, 8);
  data.writeUInt32BE(0, 12);
  data.writeUInt32BE(offset, 16);
  data.writeUInt32BE(48, 20);
  data.writeUInt32BE(Math.log2(offset), 24);
  data.writeUInt32LE(0xfeedfacf, offset);
  data.writeUInt32LE(0x0100000c, offset + 4);
  data.writeUInt32LE(0, offset + 8);
  data.writeUInt32LE(2, offset + 12);
  return data;
}

async function inventory(path) {
  const scope = new ArtifactResourceScope();
  try { return await inventoryArtifact(path, { resourceScope: scope }); }
  finally { await scope.close(); }
}

const results = [];
for (const [label, offset] of [['near', 4096], ['far', 16384]]) {
  const bytes = fat(offset);
  const directPath = join(evidenceDir, `${label}.fat`);
  await writeFile(directPath, bytes);
  const direct = await inventory(directPath);
  const directRoot = direct.nodes.find(n => n.artifact_id === direct.manifest.root_artifact_id);
  assert.equal(directRoot.format, 'mach-o-universal');
  for (const format of ['zip', 'pck']) {
    const archivePath = join(evidenceDir, `${label}.${format}`);
    if (format === 'zip') {
      const writer = new ZipWriter(new Uint8ArrayWriter());
      await writer.add('native.fat', new Uint8ArrayReader(bytes), { level: 0 });
      await writeFile(archivePath, await writer.close());
    } else {
      await writeFile(archivePath, buildGodotPck(3, [{ path: 'native.fat', data: bytes }]));
    }
    const graph = await inventory(archivePath);
    const occurrence = graph.occurrences.find(row => row.logical_path === 'native.fat');
    const node = graph.nodes.find(row => row.artifact_id === occurrence.artifact_id);
    assert.equal(occurrence.artifact_format, 'mach-o-universal');
    assert.equal(occurrence.hash_status, 'verified');
    assert.equal(node.sha256, directRoot.sha256);
    assert.equal(node.format, label === 'near' ? 'mach-o-universal' : 'unknown');
    const summary = {
      case: `${label}-${format}`, sliceOffset: offset, bytes: bytes.length,
      coverage: graph.coverage, occurrenceFormat: occurrence.artifact_format,
      occurrenceHashStatus: occurrence.hash_status, linkedNodeFormat: node.format,
      directRootNodeFormat: directRoot.format, sameContentId: node.artifact_id === directRoot.artifact_id,
      sha256: node.sha256,
    };
    results.push(summary);
    await writeFile(join(evidenceDir, `${label}-${format}.inventory.json`), JSON.stringify(graph, null, 2) + '\n');
  }
}
const report = {
  sourceRevision: execFileSync('git', ['rev-parse', 'HEAD'], { cwd: root, encoding: 'utf8' }).trim(),
  runtime: { node: process.version, platform: process.platform, architecture: process.arch },
  results,
};
await writeFile(join(evidenceDir, 'fat-node-output.json'), JSON.stringify(report, null, 2) + '\n');
console.log(JSON.stringify(report, null, 2));

Expected behavior

A content node must retain the format already established from its observed bytes. These FAT fixtures should produce mach-o-universal nodes whether the bytes appear as an archive member or as a standalone root.

This is an existing contract: the node schema defines formats from bytes independently of path roles, and the root-node producer consumes retained FAT slice evidence.

Impact and scope

Observed: format information established by the scanner is lost from two archive content nodes, and the same content ID receives different format values depending on whether it is a root or member. Consumers that read nodes[].format lose a known native format. Bytes, SHA-256, and the occurrence's format remained correct in the probes; no extraction failure or downstream consumer failure was tested.

The confirmed inputs are small synthetic ZIP and PCK v3 archives with one FAT32 member. Other readers use the shared producer, but their runtime behavior was not tested. The returned occurrence format is an available workaround for callers that have occurrence data.

REA version

Source 76b56957e94d804667f7890b244639ecc4725b32, package version 6.7.0.

Release distinction: ZIP inventory exists in published rea-agents-6.7.0 (27dc42d10249f0410904a75c0997842eb03b931c). The streamed FAT-evidence producer and Godot PCK reader in this reproduction are main-only additions absent from that tag. This contradictory occurrence/node result was reproduced on main only; published-tag runtime was not tested.

Runtime and platform

Node.js v24.22.1; Linux x64. No Hopper, Ghidra, lipo, or target execution.

Target and artifact context

Authored FAT32 container with one 48-byte arm64 Mach-O slice, stored as native.fat in ZIP and Godot PCK v3. Near control: 4,096-byte slice offset. Failing case: 16,384-byte offset.

Interface

Other: the production inventoryArtifact source workflow shared by artifact inspection.

Earliest failure stage

Artifact or evidence registration

Relevant output or evidence

Observed probe output, reduced to the relevant fields:

case       slice offset  occurrence format   linked node format  direct root format  same content ID
near-zip           4096  mach-o-universal    mach-o-universal    mach-o-universal    true
near-pck           4096  mach-o-universal    mach-o-universal    mach-o-universal    true
far-zip           16384  mach-o-universal    unknown             mach-o-universal    true
far-pck           16384  mach-o-universal    unknown             mach-o-universal    true

Every archive occurrence had hash_status: verified and every archive had coverage: {scope: container-entries, status: complete}. Both far members and their standalone roots had SHA-256 c516257f9d11f6e204b8d4430ecff5a8333bcbf011655985da4c11f7909c56a4.

Evidence provenance: observed source-workflow output. Local verification: reproduced. Reproduction completeness: complete for the stated source boundary; CLI/MCP adapter and published-release runtime coverage remain untested.

Typed outcome or recovery expected

The near and far archive nodes should both retain format: mach-o-universal; their occurrence formats, content IDs, hashes, and verified bytes should stay intact.

Analysis or suspected cause

Source-derived: the member producer passes the complete digest to classifyArtifactContent, but calls classifyArtifactBytes(digest.prefix, digest.bytes) without digest.fatSliceHeaders for the linked node. The byte classifier requires that retained evidence when a slice header lies outside the prefix. The root producer already passes it.

Additional context

Open and closed issue searches included artifact node FAT, fatSliceHeaders, mach-o-universal unknown, and inventory classification prefix. Related closed #2317 concerns class files falsely identified as FAT images; this report concerns valid FAT evidence discarded after it has been observed. No matching report was found in those searches.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Two executed archive inventories lose a byte-established format from content nodes and disagree with their own verified occurrences and identical standalone content nodes. The observed failure is bounded metadata correctness; bytes and hashes remain intact.
Workaround status: available
Workaround basis: The same returned occurrence preserves artifact_format: mach-o-universal in both failing probes, so a caller with occurrence data can use that established format without changing the artifact.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6741 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The shared artifact inventory workflow returns conflicting format facts for an ordinary ZIP member containing a synthetic FAT Mach-O image. The member occurrence is `mach-o-universal` with a verified hash, but its linked content node is `unknown`. Inspecting the identical bytes as the root produces `mach-o-universal` for the same content artifact ID. A Godot PCK v3 member produces the same mismatch. Both archive inventories report complete container-entry coverage. This was observed when the thin arm64 slice started at offset 16,384, beyond the scanner's 8,192-byte classification prefix. A 4,096-offset control produces matching `mach-o-universal` formats throughout. ## Reproduction From the source revision below with its locked dependencies installed, save the following as `$WORK/probe-fat-node.mjs`, where `$WORK` is an existing disposable directory outside the checkout. Run: ```sh node --import ./node_modules/tsx/dist/loader.mjs "$WORK/probe-fat-node.mjs" "$WORK/evidence" ``` This is the probe actually run, once for each near/far ZIP/PCK case and once per direct-root control. It only writes and inventories inert authored bytes. ```js import assert from 'node:assert/strict'; import { mkdir, writeFile } from 'node:fs/promises'; import { join, resolve } from 'node:path'; import { pathToFileURL } from 'node:url'; import { createRequire } from 'node:module'; import { execFileSync } from 'node:child_process'; const root = resolve(process.env.REA_SOURCE ?? process.cwd()); const evidenceDir = resolve(process.argv[2] ?? '.'); await mkdir(evidenceDir, { recursive: true }); const source = (path) => import(pathToFileURL(join(root, path)).href); const { inventoryArtifact } = await source('src/artifacts/inventory/ArtifactInventory.ts'); const { ArtifactResourceScope } = await source('src/artifacts/ArtifactResourceScope.ts'); const { buildGodotPck } = await source('tests/fixtures/godotPck.ts'); const require = createRequire(join(root, 'package.json')); const { ZipWriter, Uint8ArrayWriter, Uint8ArrayReader } = await import(require.resolve('@zip.js/zip.js')); function fat(offset) { const data = Buffer.alloc(offset + 48); data.writeUInt32BE(0xcafebabe, 0); data.writeUInt32BE(1, 4); data.writeUInt32BE(0x0100000c, 8); data.writeUInt32BE(0, 12); data.writeUInt32BE(offset, 16); data.writeUInt32BE(48, 20); data.writeUInt32BE(Math.log2(offset), 24); data.writeUInt32LE(0xfeedfacf, offset); data.writeUInt32LE(0x0100000c, offset + 4); data.writeUInt32LE(0, offset + 8); data.writeUInt32LE(2, offset + 12); return data; } async function inventory(path) { const scope = new ArtifactResourceScope(); try { return await inventoryArtifact(path, { resourceScope: scope }); } finally { await scope.close(); } } const results = []; for (const [label, offset] of [['near', 4096], ['far', 16384]]) { const bytes = fat(offset); const directPath = join(evidenceDir, `${label}.fat`); await writeFile(directPath, bytes); const direct = await inventory(directPath); const directRoot = direct.nodes.find(n => n.artifact_id === direct.manifest.root_artifact_id); assert.equal(directRoot.format, 'mach-o-universal'); for (const format of ['zip', 'pck']) { const archivePath = join(evidenceDir, `${label}.${format}`); if (format === 'zip') { const writer = new ZipWriter(new Uint8ArrayWriter()); await writer.add('native.fat', new Uint8ArrayReader(bytes), { level: 0 }); await writeFile(archivePath, await writer.close()); } else { await writeFile(archivePath, buildGodotPck(3, [{ path: 'native.fat', data: bytes }])); } const graph = await inventory(archivePath); const occurrence = graph.occurrences.find(row => row.logical_path === 'native.fat'); const node = graph.nodes.find(row => row.artifact_id === occurrence.artifact_id); assert.equal(occurrence.artifact_format, 'mach-o-universal'); assert.equal(occurrence.hash_status, 'verified'); assert.equal(node.sha256, directRoot.sha256); assert.equal(node.format, label === 'near' ? 'mach-o-universal' : 'unknown'); const summary = { case: `${label}-${format}`, sliceOffset: offset, bytes: bytes.length, coverage: graph.coverage, occurrenceFormat: occurrence.artifact_format, occurrenceHashStatus: occurrence.hash_status, linkedNodeFormat: node.format, directRootNodeFormat: directRoot.format, sameContentId: node.artifact_id === directRoot.artifact_id, sha256: node.sha256, }; results.push(summary); await writeFile(join(evidenceDir, `${label}-${format}.inventory.json`), JSON.stringify(graph, null, 2) + '\n'); } } const report = { sourceRevision: execFileSync('git', ['rev-parse', 'HEAD'], { cwd: root, encoding: 'utf8' }).trim(), runtime: { node: process.version, platform: process.platform, architecture: process.arch }, results, }; await writeFile(join(evidenceDir, 'fat-node-output.json'), JSON.stringify(report, null, 2) + '\n'); console.log(JSON.stringify(report, null, 2)); ``` ## Expected behavior A content node must retain the format already established from its observed bytes. These FAT fixtures should produce `mach-o-universal` nodes whether the bytes appear as an archive member or as a standalone root. This is an existing contract: [the node schema defines formats from bytes independently of path roles](https://github.com/morluto/rea/blob/76b56957e94d804667f7890b244639ecc4725b32/src/domain/artifactGraph.ts#L90), and [the root-node producer consumes retained FAT slice evidence](https://github.com/morluto/rea/blob/76b56957e94d804667f7890b244639ecc4725b32/src/artifacts/inventory/ArtifactGraphConstruction.ts#L218-L225). ## Impact and scope Observed: format information established by the scanner is lost from two archive content nodes, and the same content ID receives different format values depending on whether it is a root or member. Consumers that read `nodes[].format` lose a known native format. Bytes, SHA-256, and the occurrence's format remained correct in the probes; no extraction failure or downstream consumer failure was tested. The confirmed inputs are small synthetic ZIP and PCK v3 archives with one FAT32 member. Other readers use the shared producer, but their runtime behavior was not tested. The returned occurrence format is an available workaround for callers that have occurrence data. ## REA version Source `76b56957e94d804667f7890b244639ecc4725b32`, package version 6.7.0. Release distinction: ZIP inventory exists in published `rea-agents-6.7.0` (`27dc42d10249f0410904a75c0997842eb03b931c`). The streamed FAT-evidence producer and Godot PCK reader in this reproduction are main-only additions absent from that tag. This contradictory occurrence/node result was reproduced on main only; published-tag runtime was not tested. ## Runtime and platform Node.js v24.22.1; Linux x64. No Hopper, Ghidra, lipo, or target execution. ## Target and artifact context Authored FAT32 container with one 48-byte arm64 Mach-O slice, stored as `native.fat` in ZIP and Godot PCK v3. Near control: 4,096-byte slice offset. Failing case: 16,384-byte offset. ## Interface Other: the production `inventoryArtifact` source workflow shared by artifact inspection. ## Earliest failure stage Artifact or evidence registration ## Relevant output or evidence Observed probe output, reduced to the relevant fields: ```text case slice offset occurrence format linked node format direct root format same content ID near-zip 4096 mach-o-universal mach-o-universal mach-o-universal true near-pck 4096 mach-o-universal mach-o-universal mach-o-universal true far-zip 16384 mach-o-universal unknown mach-o-universal true far-pck 16384 mach-o-universal unknown mach-o-universal true ``` Every archive occurrence had `hash_status: verified` and every archive had `coverage: {scope: container-entries, status: complete}`. Both far members and their standalone roots had SHA-256 `c516257f9d11f6e204b8d4430ecff5a8333bcbf011655985da4c11f7909c56a4`. Evidence provenance: observed source-workflow output. Local verification: reproduced. Reproduction completeness: complete for the stated source boundary; CLI/MCP adapter and published-release runtime coverage remain untested. ## Typed outcome or recovery expected The near and far archive nodes should both retain `format: mach-o-universal`; their occurrence formats, content IDs, hashes, and verified bytes should stay intact. ## Analysis or suspected cause Source-derived: [the member producer](https://github.com/morluto/rea/blob/76b56957e94d804667f7890b244639ecc4725b32/src/artifacts/inventory/scanReader.ts#L341-L350) passes the complete digest to `classifyArtifactContent`, but calls `classifyArtifactBytes(digest.prefix, digest.bytes)` without `digest.fatSliceHeaders` for the linked node. [The byte classifier](https://github.com/morluto/rea/blob/76b56957e94d804667f7890b244639ecc4725b32/src/artifacts/inventory/ArtifactGraphConstruction.ts#L605-L630) requires that retained evidence when a slice header lies outside the prefix. The root producer already passes it. ## Additional context Open and closed issue searches included `artifact node FAT`, `fatSliceHeaders`, `mach-o-universal unknown`, and `inventory classification prefix`. Related closed [#2317](https://github.com/morluto/rea/issues/2317) concerns class files falsely identified as FAT images; this report concerns valid FAT evidence discarded after it has been observed. No matching report was found in those searches. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Two executed archive inventories lose a byte-established format from content nodes and disagree with their own verified occurrences and identical standalone content nodes. The observed failure is bounded metadata correctness; bytes and hashes remain intact. - Workaround status: available - Workaround basis: The same returned occurrence preserves `artifact_format: mach-o-universal` in both failing probes, so a caller with occurrence data can use that established format without changing the artifact.

bsakaguchi closed this issue

2026-10-31 00:46:50 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:32 +00:00

[PR #4103] [MERGED] fix(providers): report provider-selection diagnostics when the selected provider refuses its profile #6059

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425