morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #5895] [Bug] Valid safetensors header lengths are misclassified as PyTorch pickle #3575

New issue

Closed

opened 2026-10-31 00:42:01 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-31 00:42:01 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/5895

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-model-artifact rejects valid safetensors files when the JSON header length is 640, 896, 1152, or 1408 bytes. The CLI exits 1 with unsupported_target and tells the caller to supply GGUF or safetensors, although the input is already safetensors.

Adding eight trailing spaces to each header, updating its length prefix, and retaining the same tensor declaration and four payload bytes makes the corresponding inspection succeed. Both runs of the nine-case reproduction below produced the same results.

Reproduction

From the built repository root, run this script with Node. It generates temporary single-tensor files, invokes the production CLI, prints the relevant result fields, and deletes the generated files:

import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { spawnSync } from 'node:child_process';
const cli = resolve('scripts/rea.mjs');
const directory = mkdtempSync(join(tmpdir(), 'safetensors-case-'));
try {
  const header = JSON.stringify({weight: {dtype: 'F32', shape: [1], data_offsets: [0, 4]}});
  for (const length of [632, 640, 648, 896, 904, 1152, 1160, 1408, 1416]) {
    const prefix = Buffer.alloc(8);
    prefix.writeBigUInt64LE(BigInt(length));
    const path = join(directory, `model-${length}.safetensors`);
    writeFileSync(path, Buffer.concat([prefix, Buffer.from(header.padEnd(length, ' ')), Buffer.alloc(4)]));
    const result = spawnSync(process.execPath, [cli, 'inspect-model-artifact', path, '--json'], {encoding: 'utf8', timeout: 30000});
    if (result.error) throw result.error;
    const output = JSON.parse(result.stdout);
    console.log(JSON.stringify({length, prefix: prefix.toString('hex'), exit: result.status, code: output.code ?? null, reason: output.details?.reason ?? null, format: output.normalized_result?.format ?? null}));
  }
} finally {
  rmSync(directory, {recursive: true, force: true});
}

The fixtures contain one F32 tensor named weight, shape [1], offsets [0,4], and four zero payload bytes. Only the header padding and length prefix vary.

Expected behavior

All nine inputs should be recognized as safetensors and return the tensor declaration. REA documents content-based safetensors inspection.

The safetensors format definition linked by REA defines an eight-byte unsigned little-endian header length, followed by UTF-8 JSON beginning with an opening brace; trailing space padding is allowed. Nothing in that definition excludes these header lengths. For length 640, the length field is 80 02 00 00 00 00 00 00, followed by the JSON opening brace.

Impact and scope

Inspection fails before producing model Evidence for these supported inputs. The failure is deterministic in the synthetic fixtures. A demonstrated workaround is to inspect a copy with eight extra trailing header spaces and a correspondingly updated length field; tensor declarations and payload stay the same, but the header digest changes.

The code also matches lengths congruent to these values modulo 65536, subject to other reader limits; that broader range is source-derived. Real-model frequency is unknown. CLI execution was tested; MCP reachability follows the shared service composition and was not separately executed. No model was loaded or run, and no external analysis provider was used.

REA version

6.7.0; source revision 52415975942a4b9c53a817915b313ccc404f280e.

Runtime and platform

macOS 27.0.1 arm64; Node v26.12.0.

Target and artifact context

Generated valid safetensors files with one F32 tensor, four payload bytes, and space-padded JSON headers.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed result matrix:

JSON header bytes   First 8 bytes       Exit   Result
632                 7802000000000000     0     safetensors
640                 8002000000000000     1     unsupported_target
648                 8802000000000000     0     safetensors
896                 8003000000000000     1     unsupported_target
904                 8803000000000000     0     safetensors
1152                8004000000000000     1     unsupported_target
1160                8804000000000000     0     safetensors
1408                8005000000000000     1     unsupported_target
1416                8805000000000000     0     safetensors

Every failed case returned this details.reason:

PyTorch pickle/ZIP model deserialization is unsupported; supply GGUF or safetensors instead

Every successful case returned normalized_result.format: "safetensors" with the same tensor shape, dtype, and declared byte range.

Typed outcome or recovery expected

Return normal model-artifact Evidence for valid safetensors files. Keep unsupported PyTorch files refused, but distinguish them from valid safetensors length prefixes before returning unsupported_target.

Analysis or suspected cause

The provider checks for pickle/ZIP prefixes before selecting the safetensors parser. Its pickle predicate uses only the first two bytes: 80 followed by a value from 02 through 05. Those bytes also occur in valid little-endian header lengths. The safetensors reader can validate the length and opening brace, but the early refusal prevents it from running.

Additional context

Fetched and checked origin/main at c324d31b45fcab77e4ed7320ed60f0bf5a4ba188; the provider, safetensors parser, and model documentation were unchanged from the tested revision. Checked related model-inspection and PyTorch-support reports; no matching collision report found. Fixture validity was checked against the format definition; an independent reference-library reader was not run.

Originally created by @coygeek on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/5895 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior `inspect-model-artifact` rejects valid safetensors files when the JSON header length is 640, 896, 1152, or 1408 bytes. The CLI exits 1 with `unsupported_target` and tells the caller to supply GGUF or safetensors, although the input is already safetensors. Adding eight trailing spaces to each header, updating its length prefix, and retaining the same tensor declaration and four payload bytes makes the corresponding inspection succeed. Both runs of the nine-case reproduction below produced the same results. ### Reproduction From the built repository root, run this script with Node. It generates temporary single-tensor files, invokes the production CLI, prints the relevant result fields, and deletes the generated files: ```js import { mkdtempSync, writeFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join, resolve } from 'node:path'; import { spawnSync } from 'node:child_process'; const cli = resolve('scripts/rea.mjs'); const directory = mkdtempSync(join(tmpdir(), 'safetensors-case-')); try { const header = JSON.stringify({weight: {dtype: 'F32', shape: [1], data_offsets: [0, 4]}}); for (const length of [632, 640, 648, 896, 904, 1152, 1160, 1408, 1416]) { const prefix = Buffer.alloc(8); prefix.writeBigUInt64LE(BigInt(length)); const path = join(directory, `model-${length}.safetensors`); writeFileSync(path, Buffer.concat([prefix, Buffer.from(header.padEnd(length, ' ')), Buffer.alloc(4)])); const result = spawnSync(process.execPath, [cli, 'inspect-model-artifact', path, '--json'], {encoding: 'utf8', timeout: 30000}); if (result.error) throw result.error; const output = JSON.parse(result.stdout); console.log(JSON.stringify({length, prefix: prefix.toString('hex'), exit: result.status, code: output.code ?? null, reason: output.details?.reason ?? null, format: output.normalized_result?.format ?? null})); } } finally { rmSync(directory, {recursive: true, force: true}); } ``` The fixtures contain one `F32` tensor named `weight`, shape `[1]`, offsets `[0,4]`, and four zero payload bytes. Only the header padding and length prefix vary. ### Expected behavior All nine inputs should be recognized as safetensors and return the tensor declaration. [REA documents content-based safetensors inspection](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/docs/model-artifacts.md#L3-L6). The [safetensors format definition linked by REA](https://github.com/huggingface/safetensors/blob/e246a2560645b7525f5775669ed816eb57c5bcc8/README.md#L66-L90) defines an eight-byte unsigned little-endian header length, followed by UTF-8 JSON beginning with an opening brace; trailing space padding is allowed. Nothing in that definition excludes these header lengths. For length 640, the length field is `80 02 00 00 00 00 00 00`, followed by the JSON opening brace. ### Impact and scope Inspection fails before producing model Evidence for these supported inputs. The failure is deterministic in the synthetic fixtures. A demonstrated workaround is to inspect a copy with eight extra trailing header spaces and a correspondingly updated length field; tensor declarations and payload stay the same, but the header digest changes. The code also matches lengths congruent to these values modulo 65536, subject to other reader limits; that broader range is source-derived. Real-model frequency is unknown. CLI execution was tested; MCP reachability follows the shared service composition and was not separately executed. No model was loaded or run, and no external analysis provider was used. ### REA version 6.7.0; source revision `52415975942a4b9c53a817915b313ccc404f280e`. ### Runtime and platform macOS 27.0.1 arm64; Node v26.12.0. ### Target and artifact context Generated valid safetensors files with one F32 tensor, four payload bytes, and space-padded JSON headers. ### Interface CLI ### Earliest failure stage Native analysis or reconstruction ### Relevant output or evidence Observed result matrix: ```text JSON header bytes First 8 bytes Exit Result 632 7802000000000000 0 safetensors 640 8002000000000000 1 unsupported_target 648 8802000000000000 0 safetensors 896 8003000000000000 1 unsupported_target 904 8803000000000000 0 safetensors 1152 8004000000000000 1 unsupported_target 1160 8804000000000000 0 safetensors 1408 8005000000000000 1 unsupported_target 1416 8805000000000000 0 safetensors ``` Every failed case returned this `details.reason`: ```text PyTorch pickle/ZIP model deserialization is unsupported; supply GGUF or safetensors instead ``` Every successful case returned `normalized_result.format: "safetensors"` with the same tensor shape, dtype, and declared byte range. ### Typed outcome or recovery expected Return normal model-artifact Evidence for valid safetensors files. Keep unsupported PyTorch files refused, but distinguish them from valid safetensors length prefixes before returning `unsupported_target`. ### Analysis or suspected cause The [provider checks for pickle/ZIP prefixes before selecting the safetensors parser](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/model/ModelArtifactProvider.ts#L258-L267). Its pickle predicate uses only the first two bytes: `80` followed by a value from `02` through `05`. Those bytes also occur in valid little-endian header lengths. The [safetensors reader](https://github.com/morluto/rea/blob/52415975942a4b9c53a817915b313ccc404f280e/src/model/SafetensorsHeader.ts#L156-L185) can validate the length and opening brace, but the early refusal prevents it from running. ### Additional context Fetched and checked `origin/main` at `c324d31b45fcab77e4ed7320ed60f0bf5a4ba188`; the provider, safetensors parser, and model documentation were unchanged from the tested revision. Checked related model-inspection and PyTorch-support reports; no matching collision report found. Fixture validity was checked against the format definition; an independent reference-library reader was not run.

bsakaguchi closed this issue

2026-10-31 00:42:01 +00:00

bsakaguchi commented

2026-10-31 00:42:02 +00:00

Author

Owner

@amyanger commented on GitHub (Oct 21, 2026):

I'm preparing a fix for this with regression tests and will open a draft PR soon.

@amyanger commented on GitHub (Oct 21, 2026): I'm preparing a fix for this with regression tests and will open a draft PR soon.

bsakaguchi referenced this issue

2026-10-31 01:09:47 +00:00

[PR #3575] [MERGED] fix(contracts): reject NUL bytes in Frida, Rizin, and GDB local path inputs #5834

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#3575

No description provided.

Rows
Columns