[GH-ISSUE #7296] [Bug] A valid safetensors length is reported as unsupported PyTorch ZIP instead of a resource limit #4335

New issue

Closed

opened 2026-10-31 00:48:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:48:59 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/7296

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

inspect-model-artifact FILE --json reports a format-valid safetensors file with a 67,324,752-byte JSON header as unsupported_target and tells the caller to supply GGUF or safetensors. The input already is safetensors. The same declaration and payload with eight more trailing header spaces return resource_constraint for the retained representation budget. Both commands exit 1; successful inspection is unavailable at either size under that budget.

Reproduction

From the built repository root, save this script outside the checkout as model-format-probe.mjs and run node /path/to/model-format-probe.mjs with the pinned Node runtime. It writes one fixture at a time outside the checkout, isolates the CLI child environment and configuration, invokes the public CLI, and removes the fixtures. Each case was run once. The ZIP contains one inert text file; the pickle bytes encode a dictionary and are never deserialized.

import { mkdtempSync, mkdirSync, openSync, writeSync, closeSync, statSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { spawnSync } from 'node:child_process';
const cli = resolve('scripts/rea.mjs');
const directory = mkdtempSync(join(tmpdir(), 'model-format-'));
const home = join(directory, 'home');
mkdirSync(home);
const env = { PATH: process.env.PATH, HOME: home, XDG_CONFIG_HOME: home, XDG_CACHE_HOME: home, XDG_DATA_HOME: home, TMPDIR: directory };
const header = Buffer.from(JSON.stringify({weight:{dtype:'U8',shape:[1],data_offsets:[0,1]}}));
const padding = Buffer.alloc(1024 * 1024, 0x20);
function inspect(name, path, prefix) {
  const run = spawnSync(process.execPath, [cli, 'inspect-model-artifact', path, '--json'], {env, encoding:'utf8',timeout:60000,maxBuffer:2*1024*1024});
  if (run.error) throw run.error;
  const output = JSON.parse(run.stdout);
  const result = {name,bytes:statSync(path).size,prefix,exit:run.status,code:output.code??null,category:output.category??null,reason:output.details?.reason??null,operation:output.details?.operation??null,resource:output.details?.resource??null,limits:output.details?.reported_limits??null,format:output.normalized_result?.format??null,tensors:output.normalized_result?.tensors??null};
  console.log(JSON.stringify(result));
  rmSync(path);
}
try {
  for (const length of [64,640,67324752,67324760]) {
    const path = join(directory, `model-${length}.safetensors`);
    const prefix = Buffer.alloc(8); prefix.writeBigUInt64LE(BigInt(length));
    const fd = openSync(path,'wx');
    try {
      writeSync(fd,prefix); writeSync(fd,header);
      for(let remaining=length-header.length;remaining>0;) { const n=Math.min(remaining,padding.length); writeSync(fd,padding,0,n);remaining-=n; }
      writeSync(fd,Buffer.from([0x5a]));
    } finally {closeSync(fd);}
    if (statSync(path).size!==length+9) throw new Error('invalid fixture size');
    inspect(`safetensors-${length}`,path,prefix.toString('hex'));
  }
  const controls = [
    ['inert.zip','504b0304140000000000000021008bc5848d05000000050000000a00000074656e736f722e747874696e657274504b01021403140000000000000021008bc5848d05000000050000000a000000000000000000000080010000000074656e736f722e747874504b05060000000001000100380000002d0000000000'],
    ['inert.pkl','80027d710058010000006171014b01732e'],
  ];
  for(const [name,hex] of controls) {const path=join(directory,name);writeFileSync(path,Buffer.from(hex,'hex'));inspect(name,path,hex.slice(0,16));}
} finally {rmSync(directory,{recursive:true,force:true});}

Expected behavior

Recognize the admissible safetensors length and opening brace, then report the independent retained representation constraint. The length is below the documented 100,000,000-byte safetensors JSON ceiling; trailing space padding is valid. This expectation preserves the 32 MiB representation budget and the unsupported status of genuine ZIP and pickle files.

Impact and scope

The exact file receives the wrong format category and misleading recovery advice instead of the fixed-limit explanation. This is a diagnostic defect: the file would still be refused after correct classification. The small safetensors controls succeed, and genuine inert ZIP/pickle controls remain unsupported. Real-model prevalence is unknown. CLI execution was observed; MCP was not run. Changing padding changes the artifact bytes and digest and therefore does not restore the result for the original artifact.

REA version

Built source revision 0d1a454e25db825567782e461da21530bad4b158, using its existing compiled runtime and locked dependencies.

Runtime and platform

Node.js v24.22.1; macOS 27.0.1 arm64. No external analysis provider or model execution was used.

Target and artifact context

Generated safetensors files contain one U8 tensor named weight, shape [1], offsets [0,1], and one payload byte. The header starts with { and ends with space padding. Its length is encoded as an eight-byte little-endian unsigned integer. Format validity follows the pinned safetensors specification linked by REA and fixture construction; an independent reference-library reader was not run.

Interface

CLI

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Expected source: docs/model-artifacts.md:3-6,253-263 documents content-based detection, the separate header/representation budgets, typed resource refusals, and unsupported pickle/ZIP inputs; src/model/SafetensorsHeader.ts:214-243 owns the length/brace interpretation and representation charge; src/model/ModelFormat.ts:46-49,80-115 defines the representation limit; tests/boundary/model/modelArtifact.test.ts:315-337 requires admitted large safetensors headers to receive the representation refusal.
Failure source: executed public CLI results below; src/model/ModelArtifactProvider.ts:252-258 rejects ZIP magic before the safetensors length is read, and lines 121-148 map the refusal to the public error category. The matching early check is present in the compiled provider used by the CLI.
Evidence provenance: observed; the explanation of the responsible branch is source-derived.
Local verification: reproduced
Reproduction completeness: complete

Input	Header prefix	Exit	Result
Safetensors, 64-byte header	`4000000000000000`	0	`format: safetensors`, one consistent U8 tensor
Safetensors, 640-byte header	`8002000000000000`	0	`format: safetensors`, same tensor
Safetensors, 67,324,752-byte header	`504b030400000000`	1	`unsupported_target`
Safetensors, 67,324,760-byte header	`584b030400000000`	1	`resource_constraint`, `resource: memory`, `maximum_representation_bytes: 33554432`
Genuine inert ZIP	`504b030414000000`	1	`unsupported_target`
Inert protocol-2 pickle	`80027d7100580100`	1	`unsupported_target`

The colliding file reports PyTorch pickle/ZIP model deserialization is unsupported; supply GGUF or safetensors instead. The neighboring large-header control reports Model facts exceed the 32 MiB retained representation budget. This release has no runtime override. Both error category values match their respective code values and identify operation inspect_model_artifact.

Typed outcome or recovery expected

Restoration check: the exact 67,324,752-byte-header fixture should return code and category resource_constraint, operation inspect_model_artifact, resource memory, and reported_limits.maximum_representation_bytes: 33554432, with the representation-budget reason. Keep that result for the neighboring large-header control, successful tensor inspection for both small controls, and unsupported_target for the genuine ZIP and pickle controls. This checks correct classification while preserving the resource and unsupported-format boundaries.

Analysis or suspected cause

The length 67,324,752 is hexadecimal 0x04034b50, so its low four bytes match the ZIP local-file-header signature. The provider's early ZIP check prevents the reader from considering the remaining length bytes and opening brace. Source-derived reasoning predicts the representation refusal once this valid safetensors prefix reaches its reader; twice this header size exceeds the independent budget. No implementation is prescribed.

Additional context

Live open/closed issue and PR searches for safetensors, 67,324,752, 67324752, and ZIP model found no exact open duplicate. Issue #5895 is closed/completed and concerns pickle-like lengths that should successfully inspect; the 640-byte control confirms that behavior is restored here. Merged PR #5918 explicitly excludes this ZIP collision and identifies it as a follow-up. Its full body and discussions were checked. No known-good revision for this ZIP case was established.

Triage assessment

Impact level: P3
Assessment status: supported
Impact basis: The bounded effect is an incorrect refusal category and recovery explanation for one admissible header length; the independent representation budget still prevents inspection, so no successful inspection capability or valid Evidence is lost in this case.
Workaround status: none
Workaround basis: The same original bytes take the early ZIP refusal, and the representation budget has no runtime override. Adding padding demonstrates the expected diagnostic but changes the artifact bytes and digest, so it is a control rather than a workaround for the original target.

Originally created by @coygeek on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/7296 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior `inspect-model-artifact FILE --json` reports a format-valid safetensors file with a 67,324,752-byte JSON header as `unsupported_target` and tells the caller to supply GGUF or safetensors. The input already is safetensors. The same declaration and payload with eight more trailing header spaces return `resource_constraint` for the retained representation budget. Both commands exit 1; successful inspection is unavailable at either size under that budget. ## Reproduction From the built repository root, save this script outside the checkout as `model-format-probe.mjs` and run `node /path/to/model-format-probe.mjs` with the pinned Node runtime. It writes one fixture at a time outside the checkout, isolates the CLI child environment and configuration, invokes the public CLI, and removes the fixtures. Each case was run once. The ZIP contains one inert text file; the pickle bytes encode a dictionary and are never deserialized. ```js import { mkdtempSync, mkdirSync, openSync, writeSync, closeSync, statSync, writeFileSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join, resolve } from 'node:path'; import { spawnSync } from 'node:child_process'; const cli = resolve('scripts/rea.mjs'); const directory = mkdtempSync(join(tmpdir(), 'model-format-')); const home = join(directory, 'home'); mkdirSync(home); const env = { PATH: process.env.PATH, HOME: home, XDG_CONFIG_HOME: home, XDG_CACHE_HOME: home, XDG_DATA_HOME: home, TMPDIR: directory }; const header = Buffer.from(JSON.stringify({weight:{dtype:'U8',shape:[1],data_offsets:[0,1]}})); const padding = Buffer.alloc(1024 * 1024, 0x20); function inspect(name, path, prefix) { const run = spawnSync(process.execPath, [cli, 'inspect-model-artifact', path, '--json'], {env, encoding:'utf8',timeout:60000,maxBuffer:2*1024*1024}); if (run.error) throw run.error; const output = JSON.parse(run.stdout); const result = {name,bytes:statSync(path).size,prefix,exit:run.status,code:output.code??null,category:output.category??null,reason:output.details?.reason??null,operation:output.details?.operation??null,resource:output.details?.resource??null,limits:output.details?.reported_limits??null,format:output.normalized_result?.format??null,tensors:output.normalized_result?.tensors??null}; console.log(JSON.stringify(result)); rmSync(path); } try { for (const length of [64,640,67324752,67324760]) { const path = join(directory, `model-${length}.safetensors`); const prefix = Buffer.alloc(8); prefix.writeBigUInt64LE(BigInt(length)); const fd = openSync(path,'wx'); try { writeSync(fd,prefix); writeSync(fd,header); for(let remaining=length-header.length;remaining>0;) { const n=Math.min(remaining,padding.length); writeSync(fd,padding,0,n);remaining-=n; } writeSync(fd,Buffer.from([0x5a])); } finally {closeSync(fd);} if (statSync(path).size!==length+9) throw new Error('invalid fixture size'); inspect(`safetensors-${length}`,path,prefix.toString('hex')); } const controls = [ ['inert.zip','504b0304140000000000000021008bc5848d05000000050000000a00000074656e736f722e747874696e657274504b01021403140000000000000021008bc5848d05000000050000000a000000000000000000000080010000000074656e736f722e747874504b05060000000001000100380000002d0000000000'], ['inert.pkl','80027d710058010000006171014b01732e'], ]; for(const [name,hex] of controls) {const path=join(directory,name);writeFileSync(path,Buffer.from(hex,'hex'));inspect(name,path,hex.slice(0,16));} } finally {rmSync(directory,{recursive:true,force:true});} ``` ## Expected behavior Recognize the admissible safetensors length and opening brace, then report the independent retained representation constraint. The length is below the documented 100,000,000-byte safetensors JSON ceiling; trailing space padding is valid. This expectation preserves the 32 MiB representation budget and the unsupported status of genuine ZIP and pickle files. ## Impact and scope The exact file receives the wrong format category and misleading recovery advice instead of the fixed-limit explanation. This is a diagnostic defect: the file would still be refused after correct classification. The small safetensors controls succeed, and genuine inert ZIP/pickle controls remain unsupported. Real-model prevalence is unknown. CLI execution was observed; MCP was not run. Changing padding changes the artifact bytes and digest and therefore does not restore the result for the original artifact. ## REA version Built source revision `0d1a454e25db825567782e461da21530bad4b158`, using its existing compiled runtime and locked dependencies. ## Runtime and platform Node.js v24.22.1; macOS 27.0.1 arm64. No external analysis provider or model execution was used. ## Target and artifact context Generated safetensors files contain one U8 tensor named `weight`, shape `[1]`, offsets `[0,1]`, and one payload byte. The header starts with `{` and ends with space padding. Its length is encoded as an eight-byte little-endian unsigned integer. Format validity follows the [pinned safetensors specification linked by REA](https://github.com/safetensors/safetensors/blob/e246a2560645b7525f5775669ed816eb57c5bcc8/README.md#format) and fixture construction; an independent reference-library reader was not run. ## Interface CLI ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence - Expected source: `docs/model-artifacts.md:3-6,253-263` documents content-based detection, the separate header/representation budgets, typed resource refusals, and unsupported pickle/ZIP inputs; `src/model/SafetensorsHeader.ts:214-243` owns the length/brace interpretation and representation charge; `src/model/ModelFormat.ts:46-49,80-115` defines the representation limit; `tests/boundary/model/modelArtifact.test.ts:315-337` requires admitted large safetensors headers to receive the representation refusal. - Failure source: executed public CLI results below; `src/model/ModelArtifactProvider.ts:252-258` rejects ZIP magic before the safetensors length is read, and lines 121-148 map the refusal to the public error category. The matching early check is present in the compiled provider used by the CLI. - Evidence provenance: observed; the explanation of the responsible branch is source-derived. - Local verification: reproduced - Reproduction completeness: complete | Input | Header prefix | Exit | Result | | --- | --- | --- | --- | | Safetensors, 64-byte header | `4000000000000000` | 0 | `format: safetensors`, one consistent U8 tensor | | Safetensors, 640-byte header | `8002000000000000` | 0 | `format: safetensors`, same tensor | | Safetensors, 67,324,752-byte header | `504b030400000000` | 1 | `unsupported_target` | | Safetensors, 67,324,760-byte header | `584b030400000000` | 1 | `resource_constraint`, `resource: memory`, `maximum_representation_bytes: 33554432` | | Genuine inert ZIP | `504b030414000000` | 1 | `unsupported_target` | | Inert protocol-2 pickle | `80027d7100580100` | 1 | `unsupported_target` | The colliding file reports `PyTorch pickle/ZIP model deserialization is unsupported; supply GGUF or safetensors instead`. The neighboring large-header control reports `Model facts exceed the 32 MiB retained representation budget. This release has no runtime override.` Both error `category` values match their respective `code` values and identify operation `inspect_model_artifact`. ## Typed outcome or recovery expected Restoration check: the exact 67,324,752-byte-header fixture should return `code` and `category` `resource_constraint`, operation `inspect_model_artifact`, resource `memory`, and `reported_limits.maximum_representation_bytes: 33554432`, with the representation-budget reason. Keep that result for the neighboring large-header control, successful tensor inspection for both small controls, and `unsupported_target` for the genuine ZIP and pickle controls. This checks correct classification while preserving the resource and unsupported-format boundaries. ## Analysis or suspected cause The length 67,324,752 is hexadecimal `0x04034b50`, so its low four bytes match the ZIP local-file-header signature. The provider's early ZIP check prevents the reader from considering the remaining length bytes and opening brace. Source-derived reasoning predicts the representation refusal once this valid safetensors prefix reaches its reader; twice this header size exceeds the independent budget. No implementation is prescribed. ## Additional context Live open/closed issue and PR searches for `safetensors`, `67,324,752`, `67324752`, and `ZIP model` found no exact open duplicate. [Issue #5895](https://github.com/morluto/rea/issues/5895) is closed/completed and concerns pickle-like lengths that should successfully inspect; the 640-byte control confirms that behavior is restored here. [Merged PR #5918](https://github.com/morluto/rea/pull/5918) explicitly excludes this ZIP collision and identifies it as a follow-up. Its full body and discussions were checked. No known-good revision for this ZIP case was established. ## Triage assessment - Impact level: P3 - Assessment status: supported - Impact basis: The bounded effect is an incorrect refusal category and recovery explanation for one admissible header length; the independent representation budget still prevents inspection, so no successful inspection capability or valid Evidence is lost in this case. - Workaround status: none - Workaround basis: The same original bytes take the early ZIP refusal, and the representation budget has no runtime override. Adding padding demonstrates the expected diagnostic but changes the artifact bytes and digest, so it is a control rather than a workaround for the original target.

bsakaguchi closed this issue

2026-10-31 00:48:59 +00:00

bsakaguchi referenced this issue

2026-10-31 01:12:28 +00:00

[PR #4335] [MERGED] fix(setup): reject NUL bytes in registration environments #6180