morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6637] [Bug] Safetensors inspection rejects reference-reader-compatible tensor extension fields #4019

New issue

Closed

opened 2026-10-31 00:46:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:06 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6637

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

The shared ModelArtifactService.inspect boundary rejects a safetensors file containing one U8 tensor when its tensor entry includes an additional string field, "note":"local". It returns BinaryTargetError with constraint: "file_format_rejected" and says the tensor must declare dtype, shape and data_offsets, although all three are present.

The published safetensors 0.8.0 reference reader successfully deserializes the identical bytes as weight, dtype U8, shape [1], data [42]. A control differing only by omission of note succeeds in REA and the reference reader. This is a source-service reproduction; CLI and MCP transport were not executed.

Reproduction

In the source checkout below with locked dependencies available, save this as repro-extra-field.mts in an existing disposable directory:

import { writeFile } from 'node:fs/promises';
import { resolve,join } from 'node:path';
import { pathToFileURL } from 'node:url';
const [checkout,output]=process.argv.slice(2).map(x=>resolve(x));
const {ModelArtifactProvider}=await import(pathToFileURL(join(checkout,'src/model/ModelArtifactProvider.ts')).href);
const {ModelArtifactService}=await import(pathToFileURL(join(checkout,'src/application/model/ModelArtifactService.ts')).href);
const header=Buffer.from('{"weight":{"dtype":"U8","shape":[1],"data_offsets":[0,1],"note":"local"}}');
const length=Buffer.alloc(8);length.writeBigUInt64LE(BigInt(header.length));
const path=join(output,'extra-string-field.safetensors');
await writeFile(path,Buffer.concat([length,header,Buffer.from([42])]));
const service=new ModelArtifactService(new ModelArtifactProvider());
try {
 const result=await service.inspect({path,detail:'complete'});
 console.log(JSON.stringify(result.ok?{ok:true,tensors:result.value.normalized_result.tensors}:{ok:false,tag:result.error._tag,constraint:result.error.constraint,reason:result.error.reason}));
}finally {await service.close();}

Run from the checkout, supplying that directory as SCRATCH:

node node_modules/tsx/dist/cli.mjs SCRATCH/repro-extra-field.mts "$PWD" SCRATCH

This reduced driver was executed once, after an initial driver independently exercised this input and the ordinary control. Both service calls rejected the extension-field input. No engines, model execution, external references or network targets are involved.

Expected behavior

Inspect the required tensor declarations and retain their known dtype, shape and byte range when an otherwise valid entry contains an additional field accepted by the reference reader. An unsupported optional field should not make the present required declarations appear missing.

The safetensors format notes, also linked by REA's model guide, defer the JSON subset to the reference implementation. Its TensorInfo uses ordinary Serde deserialization without denying unknown fields. Successful reference-reader execution confirms acceptance for this exact fixture; this report does not request interpretation of the added field.

Impact and scope

The demonstrated file cannot be inspected through the shared model service despite having valid known tensor declarations and storage. Removing the extra field in a separately constructed copy produces the successful control, with the same U8 shape and payload. Real-world prevalence and other extension-field types were not measured. No released-version regression is claimed.

REA version

Source SHA a281455f0b83c088ea30ebbb887398888ed167fb, package version 6.7.0, unreleased main. The published rea-agents-6.7.0 tag points to 27dc42d10249f0410904a75c0997842eb03b931c and does not contain these model-reader paths; the demonstrated scope is the source revision.

Runtime and platform

Node.js v26.12.1; Linux x86_64, Omarchy. Source execution used the locked tsx/esbuild packages; the matching esbuild 0.28.2 executable was supplied through ESBUILD_BINARY_PATH from a disposable directory because the prepared checkout lacked its optional platform binary. Reference comparison: Python 3.14.8 and the published safetensors 0.8.0 Linux x86_64 wheel, extracted into disposable storage without a global installation. No Hopper or Ghidra.

Target and artifact context

Synthetic 82-byte safetensors file: eight-byte little-endian header length, a 73-byte UTF-8 JSON object, and one byte of tensor data. The tensor has dtype U8, shape [1], offsets [0,1], plus the additional string field note. There are no gaps, overlapping ranges or trailing unindexed bytes.

Interface

Other — shared application service and production file reader.

Earliest failure stage

Native analysis or reconstruction

Relevant output or evidence

Reduced driver output:

{"ok":false,"tag":"BinaryTargetError","constraint":"file_format_rejected","reason":"Safetensors tensor weight must declare dtype, shape and data_offsets"}

The reference comparison executed safetensors.deserialize(path.read_bytes()) on the same file and returned:

{"name":"weight","dtype":"U8","shape":[1],"data":[42]}

The ordinary REA control returned shape: ["1"], byte_length: 1, parameter_count: "1" and storage_validation: "consistent". Evidence provenance: observed. Local verification: reproduced. Reproduction completeness: complete for the source-service boundary; public adapter behavior remains untested.

Typed outcome or recovery expected

A successful model artifact containing the known U8 declaration. Retaining or interpreting the extension field is outside this report.

Analysis or suspected cause

Source-derived explanation: SafetensorsHeader.ts:228-238 checks Object.keys(entry).length !== 3 alongside required-key presence. Consequently, the fourth field triggers the same refusal as a missing required field, before the valid dtype, shape and range are inspected.

Additional context

Open and closed searches covered safetensors extra fields, unknown fields and the exact refusal wording. Related #5895 concerns length-prefix detection; #6627 concerns large dimensions in empty tensors. This fixture has a small ordinary shape and does not exercise either cause.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: A bounded, reference-reader-compatible local model file is refused by the shared inspection service; both the extra-field failure and ordinary control were directly observed.
Workaround status: available
Workaround basis: The separately constructed control omits only the unrecognized string field and succeeds with the identical known tensor declaration and payload; modifying the original file is unnecessary.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6637 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior The shared `ModelArtifactService.inspect` boundary rejects a safetensors file containing one U8 tensor when its tensor entry includes an additional string field, `"note":"local"`. It returns `BinaryTargetError` with `constraint: "file_format_rejected"` and says the tensor must declare `dtype`, `shape` and `data_offsets`, although all three are present. The published safetensors 0.8.0 reference reader successfully deserializes the identical bytes as `weight`, dtype `U8`, shape `[1]`, data `[42]`. A control differing only by omission of `note` succeeds in REA and the reference reader. This is a source-service reproduction; CLI and MCP transport were not executed. ## Reproduction In the source checkout below with locked dependencies available, save this as `repro-extra-field.mts` in an existing disposable directory: ```ts import { writeFile } from 'node:fs/promises'; import { resolve,join } from 'node:path'; import { pathToFileURL } from 'node:url'; const [checkout,output]=process.argv.slice(2).map(x=>resolve(x)); const {ModelArtifactProvider}=await import(pathToFileURL(join(checkout,'src/model/ModelArtifactProvider.ts')).href); const {ModelArtifactService}=await import(pathToFileURL(join(checkout,'src/application/model/ModelArtifactService.ts')).href); const header=Buffer.from('{"weight":{"dtype":"U8","shape":[1],"data_offsets":[0,1],"note":"local"}}'); const length=Buffer.alloc(8);length.writeBigUInt64LE(BigInt(header.length)); const path=join(output,'extra-string-field.safetensors'); await writeFile(path,Buffer.concat([length,header,Buffer.from([42])])); const service=new ModelArtifactService(new ModelArtifactProvider()); try { const result=await service.inspect({path,detail:'complete'}); console.log(JSON.stringify(result.ok?{ok:true,tensors:result.value.normalized_result.tensors}:{ok:false,tag:result.error._tag,constraint:result.error.constraint,reason:result.error.reason})); }finally {await service.close();} ``` Run from the checkout, supplying that directory as `SCRATCH`: ```sh node node_modules/tsx/dist/cli.mjs SCRATCH/repro-extra-field.mts "$PWD" SCRATCH ``` This reduced driver was executed once, after an initial driver independently exercised this input and the ordinary control. Both service calls rejected the extension-field input. No engines, model execution, external references or network targets are involved. ## Expected behavior Inspect the required tensor declarations and retain their known dtype, shape and byte range when an otherwise valid entry contains an additional field accepted by the reference reader. An unsupported optional field should not make the present required declarations appear missing. The [safetensors format notes](https://github.com/huggingface/safetensors/blob/e246a2560645b7525f5775669ed816eb57c5bcc8/README.md#format), also linked by REA's model guide, defer the JSON subset to the reference implementation. Its [TensorInfo](https://github.com/huggingface/safetensors/blob/e246a2560645b7525f5775669ed816eb57c5bcc8/safetensors/src/tensor.rs#L747-L758) uses ordinary Serde deserialization without denying unknown fields. Successful reference-reader execution confirms acceptance for this exact fixture; this report does not request interpretation of the added field. ## Impact and scope The demonstrated file cannot be inspected through the shared model service despite having valid known tensor declarations and storage. Removing the extra field in a separately constructed copy produces the successful control, with the same U8 shape and payload. Real-world prevalence and other extension-field types were not measured. No released-version regression is claimed. ## REA version Source SHA `a281455f0b83c088ea30ebbb887398888ed167fb`, package version `6.7.0`, unreleased main. The published `rea-agents-6.7.0` tag points to `27dc42d10249f0410904a75c0997842eb03b931c` and does not contain these model-reader paths; the demonstrated scope is the source revision. ## Runtime and platform Node.js v26.12.1; Linux x86_64, Omarchy. Source execution used the locked tsx/esbuild packages; the matching esbuild 0.28.2 executable was supplied through `ESBUILD_BINARY_PATH` from a disposable directory because the prepared checkout lacked its optional platform binary. Reference comparison: Python 3.14.8 and the published safetensors 0.8.0 Linux x86_64 wheel, extracted into disposable storage without a global installation. No Hopper or Ghidra. ## Target and artifact context Synthetic 82-byte safetensors file: eight-byte little-endian header length, a 73-byte UTF-8 JSON object, and one byte of tensor data. The tensor has dtype U8, shape `[1]`, offsets `[0,1]`, plus the additional string field `note`. There are no gaps, overlapping ranges or trailing unindexed bytes. ## Interface Other — shared application service and production file reader. ## Earliest failure stage Native analysis or reconstruction ## Relevant output or evidence Reduced driver output: ```json {"ok":false,"tag":"BinaryTargetError","constraint":"file_format_rejected","reason":"Safetensors tensor weight must declare dtype, shape and data_offsets"} ``` The reference comparison executed `safetensors.deserialize(path.read_bytes())` on the same file and returned: ```json {"name":"weight","dtype":"U8","shape":[1],"data":[42]} ``` The ordinary REA control returned `shape: ["1"]`, `byte_length: 1`, `parameter_count: "1"` and `storage_validation: "consistent"`. Evidence provenance: observed. Local verification: reproduced. Reproduction completeness: complete for the source-service boundary; public adapter behavior remains untested. ## Typed outcome or recovery expected A successful model artifact containing the known U8 declaration. Retaining or interpreting the extension field is outside this report. ## Analysis or suspected cause Source-derived explanation: [SafetensorsHeader.ts:228-238](https://github.com/morluto/rea/blob/a281455f0b83c088ea30ebbb887398888ed167fb/src/model/SafetensorsHeader.ts#L228-L238) checks `Object.keys(entry).length !== 3` alongside required-key presence. Consequently, the fourth field triggers the same refusal as a missing required field, before the valid dtype, shape and range are inspected. ## Additional context Open and closed searches covered safetensors extra fields, unknown fields and the exact refusal wording. Related [#5895](https://github.com/morluto/rea/issues/5895) concerns length-prefix detection; [#6627](https://github.com/morluto/rea/issues/6627) concerns large dimensions in empty tensors. This fixture has a small ordinary shape and does not exercise either cause. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: A bounded, reference-reader-compatible local model file is refused by the shared inspection service; both the extra-field failure and ordinary control were directly observed. - Workaround status: available - Workaround basis: The separately constructed control omits only the unrecognized string field and succeeds with the identical known tensor declaration and payload; modifying the original file is unnecessary.

bsakaguchi closed this issue

2026-10-31 00:46:06 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:01 +00:00

[PR #4019] [CLOSED] fix(apple): stop reading a selected UID root as a keyed-archive reference #5993