morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6627] [Bug] Valid empty safetensors shapes are rejected by the file-offset precision limit #4013

New issue

Closed

opened 2026-10-31 00:46:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:03 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6627

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The production ModelArtifactService.inspect({path, detail: "complete"}) rejects an 83-byte safetensors file containing one empty F32 tensor with shape [0,9007199254740992]. It returns AnalysisResourceConstraintError with Safetensors integer exceeds exact host offset precision and maximum_exact_offset: 9007199254740991, although every actual file/data offset is below 84.

An otherwise identical 83-byte control with shape [0,9007199254740991] succeeds. The official safetensors 0.8.0 deserialize reader accepts both inputs and returns their exact shapes and zero data bytes. These are observed service-boundary results; CLI/MCP execution was not tested.

Reproduction

From the source checkout with locked dependencies installed, save this as shape-probe.mts in the checkout root and run node --import tsx shape-probe.mts with the pinned Node version. The investigation ran this service call from an external scratch probe rather than creating a file in the checkout.

import { writeFile } from "node:fs/promises";
import { resolve } from "node:path";
import { ModelArtifactProvider } from "./src/model/ModelArtifactProvider.ts";
import { ModelArtifactService } from "./src/application/model/ModelArtifactService.ts";

const path = resolve("empty-shape.safetensors");
const header = Buffer.from(
  '{"empty":{"dtype":"F32","shape":[0,9007199254740992],"data_offsets":[0,0]}}',
);
const prefix = Buffer.alloc(8);
prefix.writeBigUInt64LE(BigInt(header.length));
await writeFile(path, Buffer.concat([prefix, header]));
const provider = new ModelArtifactProvider();
try {
  const result = await new ModelArtifactService(provider).inspect({path, detail: "complete"});
  console.log(result.ok ? result.value.normalized_result : result.error.message);
} finally {
  await provider.close();
}

This produced the reported refusal on the single failing-input run. The preceding control succeeded on its single run. Replacing the large dimension with 9007199254740991 is the control, not a result-preserving workaround.

Expected behavior

Retain the shape as ["0","9007199254740992"], with zero parameters, zero tensor bytes and consistent storage, while keeping host-offset bounds on actual lengths/offsets.

REA documents empty-tensor support, and its domain stores shape dimensions as decimal strings. The upstream format implementation uses Vec<usize> for shape dimensions; on the tested 64-bit platform, the dimension fits and the zero extent makes its checked parameter/storage products zero. Acceptance by official safetensors 0.8.0 was observed directly.

Impact and scope

A bounded compatibility failure prevents declaration inspection of valid empty tensors with a dimension above JavaScript's safe-integer limit. It does not require a large file or a large allocation. Ordinary dimensions were not observed to fail. No frequency in real model collections or regression range was established. No input file was modified by REA.

REA version

6.7.0 source revision 00a9e95de91a4bd4d2a41f8aee704a53fa575750.

Runtime and platform

Node.js v24.22.1; Linux x86_64. Oracle: official safetensors 0.8.0, 64-bit Python wheel. No inference runtime or provider was involved.

Target and artifact context

Locally generated safetensors file; one F32 tensor, shape [0,9007199254740992], data_offsets: [0,0]; 75-byte JSON header plus eight-byte length prefix; no payload.

Interface

Other

Production application service and model artifact provider.

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Observed reduced results:

{"input_shape":[0,9007199254740991],"file_bytes":83,"rea_ok":true,"parameters":"0","tensor_bytes":0}
{"input_shape":[0,9007199254740992],"file_bytes":83,"rea_ok":false,"tag":"AnalysisResourceConstraintError","message":"Resource constraint during inspect_model_artifact: Safetensors integer exceeds exact host offset precision","maximum_exact_offset":9007199254740991}
{"safetensors_version":"0.8.0","input_shape":[0,9007199254740992],"accepted":true,"returned_shape":[0,9007199254740992],"data_length":0}

The first two results came from ModelArtifactService using the source revision above; the third came from safetensors.deserialize on the same saved failing fixture. The initial investigation retained the full probe and output separately. No inference or full test suite was run.

Typed outcome or recovery expected

Successful model declaration Evidence with the exact shape. The actual file offsets remain small and should not trigger maximum_exact_offset recovery.

Analysis or suspected cause

Source-derived: HeaderJson converts every integer to number and applies Number.isSafeInteger before it knows the field. Consequently, it rejects this shape before the zero-extent and BigInt parameter calculation can handle it. The scalar token/shape representation applies an offset-specific bound to dimension values.

Additional context

Open and closed issue searches for GGUF, safetensors, safetensors shape, safetensors precision, and safetensors 9007199254740992 found no matching bug report. Related feature requests concern graph/package inspection rather than this supported header parser. Last known good: NOT_ENOUGH_INFO; not tested.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A directly observed 83-byte valid safetensors input is refused, preventing its supported declaration inspection; the otherwise identical safe-integer control succeeds. The demonstrated scope is limited to empty shapes above the numeric token bound.
Workaround status: unverified
Workaround basis: The official reader successfully inspects the original declaration, but no path preserving the same REA Evidence workflow was tested. Lowering the dimension changes the input and does not preserve its declarations.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6627 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The production `ModelArtifactService.inspect({path, detail: "complete"})` rejects an 83-byte safetensors file containing one empty `F32` tensor with shape `[0,9007199254740992]`. It returns `AnalysisResourceConstraintError` with `Safetensors integer exceeds exact host offset precision` and `maximum_exact_offset: 9007199254740991`, although every actual file/data offset is below 84. An otherwise identical 83-byte control with shape `[0,9007199254740991]` succeeds. The official safetensors 0.8.0 `deserialize` reader accepts both inputs and returns their exact shapes and zero data bytes. These are observed service-boundary results; CLI/MCP execution was not tested. ## Reproduction From the source checkout with locked dependencies installed, save this as `shape-probe.mts` in the checkout root and run `node --import tsx shape-probe.mts` with the pinned Node version. The investigation ran this service call from an external scratch probe rather than creating a file in the checkout. ```ts import { writeFile } from "node:fs/promises"; import { resolve } from "node:path"; import { ModelArtifactProvider } from "./src/model/ModelArtifactProvider.ts"; import { ModelArtifactService } from "./src/application/model/ModelArtifactService.ts"; const path = resolve("empty-shape.safetensors"); const header = Buffer.from( '{"empty":{"dtype":"F32","shape":[0,9007199254740992],"data_offsets":[0,0]}}', ); const prefix = Buffer.alloc(8); prefix.writeBigUInt64LE(BigInt(header.length)); await writeFile(path, Buffer.concat([prefix, header])); const provider = new ModelArtifactProvider(); try { const result = await new ModelArtifactService(provider).inspect({path, detail: "complete"}); console.log(result.ok ? result.value.normalized_result : result.error.message); } finally { await provider.close(); } ``` This produced the reported refusal on the single failing-input run. The preceding control succeeded on its single run. Replacing the large dimension with `9007199254740991` is the control, not a result-preserving workaround. ## Expected behavior Retain the shape as `["0","9007199254740992"]`, with zero parameters, zero tensor bytes and consistent storage, while keeping host-offset bounds on actual lengths/offsets. REA documents [empty-tensor support](https://github.com/morluto/rea/blob/00a9e95de91a4bd4d2a41f8aee704a53fa575750/docs/model-artifacts.md#L61-L73), and its domain [stores shape dimensions as decimal strings](https://github.com/morluto/rea/blob/00a9e95de91a4bd4d2a41f8aee704a53fa575750/src/domain/model/modelArtifact.ts#L36-L52). The upstream format implementation uses [`Vec<usize>` for shape dimensions](https://github.com/huggingface/safetensors/blob/e246a2560645b7525f5775669ed816eb57c5bcc8/safetensors/src/tensor.rs#L795-L806); on the tested 64-bit platform, the dimension fits and the zero extent makes its checked parameter/storage products zero. Acceptance by official safetensors 0.8.0 was observed directly. ## Impact and scope A bounded compatibility failure prevents declaration inspection of valid empty tensors with a dimension above JavaScript's safe-integer limit. It does not require a large file or a large allocation. Ordinary dimensions were not observed to fail. No frequency in real model collections or regression range was established. No input file was modified by REA. ## REA version 6.7.0 source revision `00a9e95de91a4bd4d2a41f8aee704a53fa575750`. ## Runtime and platform Node.js v24.22.1; Linux x86_64. Oracle: official safetensors 0.8.0, 64-bit Python wheel. No inference runtime or provider was involved. ## Target and artifact context Locally generated safetensors file; one `F32` tensor, shape `[0,9007199254740992]`, `data_offsets: [0,0]`; 75-byte JSON header plus eight-byte length prefix; no payload. ## Interface Other Production application service and model artifact provider. ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Observed reduced results: ```json {"input_shape":[0,9007199254740991],"file_bytes":83,"rea_ok":true,"parameters":"0","tensor_bytes":0} {"input_shape":[0,9007199254740992],"file_bytes":83,"rea_ok":false,"tag":"AnalysisResourceConstraintError","message":"Resource constraint during inspect_model_artifact: Safetensors integer exceeds exact host offset precision","maximum_exact_offset":9007199254740991} {"safetensors_version":"0.8.0","input_shape":[0,9007199254740992],"accepted":true,"returned_shape":[0,9007199254740992],"data_length":0} ``` The first two results came from `ModelArtifactService` using the source revision above; the third came from `safetensors.deserialize` on the same saved failing fixture. The initial investigation retained the full probe and output separately. No inference or full test suite was run. ## Typed outcome or recovery expected Successful model declaration Evidence with the exact shape. The actual file offsets remain small and should not trigger `maximum_exact_offset` recovery. ## Analysis or suspected cause Source-derived: [`HeaderJson` converts every integer to `number` and applies `Number.isSafeInteger` before it knows the field](https://github.com/morluto/rea/blob/00a9e95de91a4bd4d2a41f8aee704a53fa575750/src/model/SafetensorsHeader.ts#L102-L116). Consequently, it rejects this shape before the [zero-extent and BigInt parameter calculation](https://github.com/morluto/rea/blob/00a9e95de91a4bd4d2a41f8aee704a53fa575750/src/model/SafetensorsHeader.ts#L242-L274) can handle it. The scalar token/shape representation applies an offset-specific bound to dimension values. ## Additional context Open and closed issue searches for `GGUF`, `safetensors`, `safetensors shape`, `safetensors precision`, and `safetensors 9007199254740992` found no matching bug report. Related feature requests concern graph/package inspection rather than this supported header parser. Last known good: NOT_ENOUGH_INFO; not tested. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A directly observed 83-byte valid safetensors input is refused, preventing its supported declaration inspection; the otherwise identical safe-integer control succeeds. The demonstrated scope is limited to empty shapes above the numeric token bound. - Workaround status: unverified - Workaround basis: The official reader successfully inspects the original declaration, but no path preserving the same REA Evidence workflow was tested. Lowering the dimension changes the input and does not preserve its declarations.

bsakaguchi closed this issue

2026-10-31 00:46:03 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:15 +00:00

[PR #4053] [CLOSED] fix(reference): preserve imports with symlinks into unreadable subtrees #6023

bsakaguchi referenced this issue

2026-10-31 01:11:17 +00:00

[PR #4060] [CLOSED] fix(reference-source): keep a partial graph for symlinks into unenumerated directories #6027

bsakaguchi referenced this issue

2026-11-01 21:43:19 +00:00

[PR #5230] fix(artifacts): keep readable entries of a directory with unreadable entries #6825