morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #6684] [Bug] Cancelling a Safetensors full hash discards the already parsed header #4060

New issue

Closed

opened 2026-10-31 00:46:28 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:46:28 +00:00

Owner

Originally created by @coygeek on GitHub (Oct 23, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/6684

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

Observed through the real source ModelArtifactService and ModelArtifactProvider, with a controlled caller cancellation after an actual tensor-data read. A valid Safetensors file has two metadata entries and one F32 tensor. Header-only inspection succeeds. When full-file hashing is cancelled after its first 1 MiB read, the service returns AnalysisCancelledError with the exact caller reason and closes the descriptor, but error.partialObservation is absent. The already parsed metadata and tensor declarations are lost from the result.

Two distinct valid Safetensors fixtures reproduced the omission, once each. An equivalent reference-generated GGUF control retained complete header/metadata/tensor observations under the same interruption. This is source-boundary execution; compiled CLI and MCP transport behavior were not tested.

Reproduction

Use source revision below with its locked dependencies prepared. Save the following as /tmp/rea-model-cancel/repro.mts and run from the checkout root:

node --import ./node_modules/tsx/dist/loader.mjs \
  /tmp/rea-model-cancel/repro.mts "$PWD" /tmp/rea-model-cancel/input

This is the reduced script actually executed. It creates a benign 2 MiB F32 fixture and calls the shared application service. The wrapper delegates every read to the actual opened FileHandle; it only observes read positions and signals cancellation after hashing starts. It does not replace provider results or inject an I/O failure.

import assert from 'node:assert/strict';
import fs from 'node:fs/promises';
import { syncBuiltinESMExports } from 'node:module';
import { resolve, join } from 'node:path';
const [checkout, root] = process.argv.slice(2).map(x => resolve(x));
await fs.mkdir(root, {recursive:true});
const path = join(root, 'model.safetensors');
const metadata = {format:'pt', description:'Benign cancellation fixture'};
const json = JSON.stringify({__metadata__:metadata,
  weight:{dtype:'F32',shape:[524288],data_offsets:[0,2097152]}});
const header = Buffer.from(json + ' '.repeat((-Buffer.byteLength(json)) & 7));
const prefix = Buffer.alloc(8); prefix.writeBigUInt64LE(BigInt(header.length));
await fs.writeFile(path, Buffer.concat([prefix,header,Buffer.alloc(2097152)]));
const { ModelArtifactProvider } = await import(join(checkout,'src/model/ModelArtifactProvider.ts'));
const { ModelArtifactService } = await import(join(checkout,'src/application/model/ModelArtifactService.ts'));
const service = new ModelArtifactService(new ModelArtifactProvider());
const originalOpen = fs.open;
try {
  const control = await service.inspect({path,hash:'header',detail:'complete'});
  assert.equal(control.ok,true);
  const report = control.value.normalized_result;
  const controller = new AbortController();
  const reason = {kind:'cancel-after-first-weight-read'};
  const reads = [];
  const handles = [];
  fs.open = async (...args) => {
    const handle = await originalOpen(...args);
    if(String(args[0])===path) {
      handles.push(handle);
      const read = handle.read.bind(handle);
      handle.read = async (...params) => {
        const result = await read(...params);
        reads.push({position:params[3],bytesRead:result.bytesRead});
        if(params[3]===report.artifact.header_length && result.bytesRead>0)
          controller.abort(reason);
        return result;
      };
    }
    return handle;
  };
  syncBuiltinESMExports();
  const result = await service.inspect({path,hash:'full',detail:'complete'},
                                       {signal:controller.signal});
  assert.equal(result.ok,false);
  assert.equal(result.error._tag,'AnalysisCancelledError');
  assert.equal(result.error.cause,reason);
  assert.equal(handles.every(h=>h.fd===-1),true);
  console.log(JSON.stringify({
    node:process.version,platform:process.platform,arch:process.arch,
    header_control:{header_length:report.artifact.header_length,
      metadata:report.metadata.length,tensors:report.tensors.length},
    cancelled:{tag:result.error._tag,exact_reason:result.error.cause===reason,
      has_partial_observation:result.error.partialObservation!==undefined,
      descriptors_closed:handles.every(h=>h.fd===-1),reads}
  },null,2));
} finally {
  fs.open = originalOpen; syncBuiltinESMExports();
  await service.close();
}

The script exited 0. Its assertions characterize cancellation, exact reason retention and confirmed descriptor closure; the printed has_partial_observation: false is the defect. The exact generated fixture was also accepted by safetensors 0.8.0 using safe_open(..., framework="numpy"), returning metadata format=pt, the benign description, key weight, dtype F32 and shape [524288].

Expected behavior

Retain completed Safetensors metadata and tensor declarations under typed cancellation, with final file identity and unfinished full-file digest explicitly unknown. The lifecycle contract requires retaining collected observations under typed cancellation. The equivalent GGUF control already does so. This expectation does not require interpreting weights or treating an interrupted inspection as success.

Impact and scope

P2: interrupted full hashing discards collected header evidence for the demonstrated supported format. This is bounded to interrupted observations; successful header-only and full-hash inspections were observed on a reference-generated Safetensors fixture. Header-only inspection is an available workaround when declarations are sufficient. Real-model interruption frequency, other platforms and public adapter propagation are NOT_ENOUGH_INFO; no damage to the input file or model execution was observed.

REA version

Source 8c4f96ef4759fd78da66c4b9726825ca71d7ee0c on main; package metadata 6.7.0. This is a main-only finding. The checked latest published tag rea-agents-6.7.0 resolves to 27dc42d10249f0410904a75c0997842eb03b931c and has no src/model, src/application/model or src/domain/model implementation. No released-version or installed-CLI reproduction is claimed.

Runtime and platform

Node.js v26.12.1; Linux x86_64; repository locked dependencies and tsx loader. Fixture cross-check: Python 3.14.8, current published safetensors 0.8.0 and gguf 0.19.0 in a scratch virtual environment. The repository's pinned Node 24.22.1 was not exercised. No model runtime or external analysis engine was used.

Target and artifact context

Generated Safetensors with two string metadata values and one F32 tensor named weight, shape [524288], offsets [0,2097152]. The reduced fixture's length-prefixed JSON header occupies 160 bytes; payload consists of zeros. A second fixture came from safetensors.numpy.save_file. Both were accepted by the reference reader. An equivalent GGUF control came from the official Python writer. No target or model code ran.

Interface

Other — real source shared application/provider boundary.

Earliest failure stage

Cleanup, cancellation, or lifecycle

Relevant output or evidence

Selected fields from the reduced script's exact stdout:

{
  "header_control": {
    "header_length": 160,
    "metadata": 2,
    "tensors": 1
  },
  "cancelled": {
    "tag": "AnalysisCancelledError",
    "exact_reason": true,
    "has_partial_observation": false,
    "descriptors_closed": true,
    "reads": [
      {
        "position": 0,
        "bytesRead": 4
      },
      {
        "position": 4,
        "bytesRead": 4
      },
      {
        "position": 8,
        "bytesRead": 1
      },
      {
        "position": 9,
        "bytesRead": 151
      },
      {
        "position": 160,
        "bytesRead": 1048576
      }
    ]
  }
}

The final read begins at the established header end and consumes 1,048,576 bytes before the caller aborts. The provider calls hashing only after Safetensors parsing and interval validation. Thus cancellation occurred after the declarations had been parsed, rather than before file admission or during JSON parsing.

In the reference-generated controls, Safetensors header-only inspection returned two metadata rows, one tensor and 2,097,152 tensor bytes. Its successful full hash matched an independent SHA-256 of the fixture. The equivalent cancelled GGUF inspection returned kind: "model-artifact-partial", complete header/metadata/tensor coverage, two metadata rows and one tensor; cancelled Safetensors returned no partial observation.

Evidence provenance: observed source-boundary execution, with source-derived causal explanation.
Local verification: reproduced.
Reproduction completeness: complete for the shared application/provider contract; public CLI and MCP remain unverified.

Typed outcome or recovery expected

Keep AnalysisCancelledError, the exact caller reason and confirmed descriptor closure. Attach the completed declarations as explicitly unfinalized partial Evidence, without asserting final file identity or an unfinished full-file SHA-256. Header-only inspection can recover declarations from an unchanged accessible file.

Analysis or suspected cause

Source-derived: partial progress is initialized only when format === "gguf". Completed Safetensors parsing is followed by reader.finish(...) without retaining its result. If hashing throws cancellation, the catch path can preserve only a completed physical operation or registered partial progress; neither exists for this Safetensors path. The partial contract is also GGUF-specific.

Additional context

Searches covered open and closed Safetensors cancellation/partial/hash reports and model hashing reports. Related closed #5625 concerned GGUF cancellation while parsing a later metadata value; its report explicitly did not establish equivalent Safetensors row loss. The GGUF control now retains its observations. Closed #6579 concerned the alignment field inside existing GGUF partial Evidence; the present Safetensors result has no partial Evidence at all. Closed #6095 concerned a committed CLI evidence-bundle receipt after publication, whereas this cancellation happens during hashing before Evidence publication.

No source files were changed. No full build, full suite, compiled CLI or public MCP cancellation was run. No claim is made about natural interruption frequency, a known-good Safetensors revision, or other failure stages.

Triage assessment

Impact level: P2
Assessment status: supported
Impact basis: Two valid local Safetensors fixtures lost all parsed header observations when full hashing was interrupted after a real payload read; exact cancellation and descriptor cleanup remained correct. Successful inspections and the equivalent GGUF partial-result control delimit the failure.
Workaround status: available
Workaround basis: Header-only source-service inspections of both fixtures succeeded and returned the same declarations without entering payload hashing. The workaround requires the file to remain available and does not provide a full-file digest.

Originally created by @coygeek on GitHub (Oct 23, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/6684 ## Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ## Area Evidence, artifacts, or reconstruction ## Observed behavior Observed through the real source `ModelArtifactService` and `ModelArtifactProvider`, with a controlled caller cancellation after an actual tensor-data read. A valid Safetensors file has two metadata entries and one F32 tensor. Header-only inspection succeeds. When full-file hashing is cancelled after its first 1 MiB read, the service returns `AnalysisCancelledError` with the exact caller reason and closes the descriptor, but `error.partialObservation` is absent. The already parsed metadata and tensor declarations are lost from the result. Two distinct valid Safetensors fixtures reproduced the omission, once each. An equivalent reference-generated GGUF control retained complete header/metadata/tensor observations under the same interruption. This is source-boundary execution; compiled CLI and MCP transport behavior were not tested. ## Reproduction Use source revision below with its locked dependencies prepared. Save the following as `/tmp/rea-model-cancel/repro.mts` and run from the checkout root: ```sh node --import ./node_modules/tsx/dist/loader.mjs \ /tmp/rea-model-cancel/repro.mts "$PWD" /tmp/rea-model-cancel/input ``` This is the reduced script actually executed. It creates a benign 2 MiB F32 fixture and calls the shared application service. The wrapper delegates every read to the actual opened FileHandle; it only observes read positions and signals cancellation after hashing starts. It does not replace provider results or inject an I/O failure. ```ts import assert from 'node:assert/strict'; import fs from 'node:fs/promises'; import { syncBuiltinESMExports } from 'node:module'; import { resolve, join } from 'node:path'; const [checkout, root] = process.argv.slice(2).map(x => resolve(x)); await fs.mkdir(root, {recursive:true}); const path = join(root, 'model.safetensors'); const metadata = {format:'pt', description:'Benign cancellation fixture'}; const json = JSON.stringify({__metadata__:metadata, weight:{dtype:'F32',shape:[524288],data_offsets:[0,2097152]}}); const header = Buffer.from(json + ' '.repeat((-Buffer.byteLength(json)) & 7)); const prefix = Buffer.alloc(8); prefix.writeBigUInt64LE(BigInt(header.length)); await fs.writeFile(path, Buffer.concat([prefix,header,Buffer.alloc(2097152)])); const { ModelArtifactProvider } = await import(join(checkout,'src/model/ModelArtifactProvider.ts')); const { ModelArtifactService } = await import(join(checkout,'src/application/model/ModelArtifactService.ts')); const service = new ModelArtifactService(new ModelArtifactProvider()); const originalOpen = fs.open; try { const control = await service.inspect({path,hash:'header',detail:'complete'}); assert.equal(control.ok,true); const report = control.value.normalized_result; const controller = new AbortController(); const reason = {kind:'cancel-after-first-weight-read'}; const reads = []; const handles = []; fs.open = async (...args) => { const handle = await originalOpen(...args); if(String(args[0])===path) { handles.push(handle); const read = handle.read.bind(handle); handle.read = async (...params) => { const result = await read(...params); reads.push({position:params[3],bytesRead:result.bytesRead}); if(params[3]===report.artifact.header_length && result.bytesRead>0) controller.abort(reason); return result; }; } return handle; }; syncBuiltinESMExports(); const result = await service.inspect({path,hash:'full',detail:'complete'}, {signal:controller.signal}); assert.equal(result.ok,false); assert.equal(result.error._tag,'AnalysisCancelledError'); assert.equal(result.error.cause,reason); assert.equal(handles.every(h=>h.fd===-1),true); console.log(JSON.stringify({ node:process.version,platform:process.platform,arch:process.arch, header_control:{header_length:report.artifact.header_length, metadata:report.metadata.length,tensors:report.tensors.length}, cancelled:{tag:result.error._tag,exact_reason:result.error.cause===reason, has_partial_observation:result.error.partialObservation!==undefined, descriptors_closed:handles.every(h=>h.fd===-1),reads} },null,2)); } finally { fs.open = originalOpen; syncBuiltinESMExports(); await service.close(); } ``` The script exited 0. Its assertions characterize cancellation, exact reason retention and confirmed descriptor closure; the printed `has_partial_observation: false` is the defect. The exact generated fixture was also accepted by `safetensors 0.8.0` using `safe_open(..., framework="numpy")`, returning metadata `format=pt`, the benign description, key `weight`, dtype `F32` and shape `[524288]`. ## Expected behavior Retain completed Safetensors metadata and tensor declarations under typed cancellation, with final file identity and unfinished full-file digest explicitly unknown. The [lifecycle contract](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/docs/tool-design.md#L129-L136) requires retaining collected observations under typed cancellation. The equivalent GGUF control already does so. This expectation does not require interpreting weights or treating an interrupted inspection as success. ## Impact and scope P2: interrupted full hashing discards collected header evidence for the demonstrated supported format. This is bounded to interrupted observations; successful header-only and full-hash inspections were observed on a reference-generated Safetensors fixture. Header-only inspection is an available workaround when declarations are sufficient. Real-model interruption frequency, other platforms and public adapter propagation are NOT_ENOUGH_INFO; no damage to the input file or model execution was observed. ## REA version Source `8c4f96ef4759fd78da66c4b9726825ca71d7ee0c` on main; package metadata 6.7.0. This is a main-only finding. The checked latest published tag `rea-agents-6.7.0` resolves to `27dc42d10249f0410904a75c0997842eb03b931c` and has no `src/model`, `src/application/model` or `src/domain/model` implementation. No released-version or installed-CLI reproduction is claimed. ## Runtime and platform Node.js v26.12.1; Linux x86_64; repository locked dependencies and tsx loader. Fixture cross-check: Python 3.14.8, current published safetensors 0.8.0 and gguf 0.19.0 in a scratch virtual environment. The repository's pinned Node 24.22.1 was not exercised. No model runtime or external analysis engine was used. ## Target and artifact context Generated Safetensors with two string metadata values and one F32 tensor named `weight`, shape `[524288]`, offsets `[0,2097152]`. The reduced fixture's length-prefixed JSON header occupies 160 bytes; payload consists of zeros. A second fixture came from `safetensors.numpy.save_file`. Both were accepted by the reference reader. An equivalent GGUF control came from the official Python writer. No target or model code ran. ## Interface Other — real source shared application/provider boundary. ## Earliest failure stage Cleanup, cancellation, or lifecycle ## Relevant output or evidence Selected fields from the reduced script's exact stdout: ```json { "header_control": { "header_length": 160, "metadata": 2, "tensors": 1 }, "cancelled": { "tag": "AnalysisCancelledError", "exact_reason": true, "has_partial_observation": false, "descriptors_closed": true, "reads": [ { "position": 0, "bytesRead": 4 }, { "position": 4, "bytesRead": 4 }, { "position": 8, "bytesRead": 1 }, { "position": 9, "bytesRead": 151 }, { "position": 160, "bytesRead": 1048576 } ] } } ``` The final read begins at the established header end and consumes 1,048,576 bytes before the caller aborts. [The provider calls hashing only after Safetensors parsing and interval validation](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/src/model/ModelArtifactProvider.ts#L313-L355). Thus cancellation occurred after the declarations had been parsed, rather than before file admission or during JSON parsing. In the reference-generated controls, Safetensors header-only inspection returned two metadata rows, one tensor and 2,097,152 tensor bytes. Its successful full hash matched an independent SHA-256 of the fixture. The equivalent cancelled GGUF inspection returned `kind: "model-artifact-partial"`, complete header/metadata/tensor coverage, two metadata rows and one tensor; cancelled Safetensors returned no partial observation. - Evidence provenance: observed source-boundary execution, with source-derived causal explanation. - Local verification: reproduced. - Reproduction completeness: complete for the shared application/provider contract; public CLI and MCP remain unverified. ## Typed outcome or recovery expected Keep `AnalysisCancelledError`, the exact caller reason and confirmed descriptor closure. Attach the completed declarations as explicitly unfinalized partial Evidence, without asserting final file identity or an unfinished full-file SHA-256. Header-only inspection can recover declarations from an unchanged accessible file. ## Analysis or suspected cause Source-derived: [partial progress is initialized only when `format === "gguf"`](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/src/model/ModelArtifactProvider.ts#L279-L312). Completed Safetensors parsing is followed by `reader.finish(...)` without retaining its result. If hashing throws cancellation, [the catch path can preserve only a completed physical operation or registered partial progress](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/src/model/ModelArtifactProvider.ts#L223-L255); neither exists for this Safetensors path. The [partial contract is also GGUF-specific](https://github.com/morluto/rea/blob/8c4f96ef4759fd78da66c4b9726825ca71d7ee0c/src/domain/model/modelArtifact.ts#L94-L123). ## Additional context Searches covered open and closed Safetensors cancellation/partial/hash reports and model hashing reports. Related closed [#5625](https://github.com/morluto/rea/issues/5625) concerned GGUF cancellation while parsing a later metadata value; its report explicitly did not establish equivalent Safetensors row loss. The GGUF control now retains its observations. Closed [#6579](https://github.com/morluto/rea/issues/6579) concerned the alignment field inside existing GGUF partial Evidence; the present Safetensors result has no partial Evidence at all. Closed [#6095](https://github.com/morluto/rea/issues/6095) concerned a committed CLI evidence-bundle receipt after publication, whereas this cancellation happens during hashing before Evidence publication. No source files were changed. No full build, full suite, compiled CLI or public MCP cancellation was run. No claim is made about natural interruption frequency, a known-good Safetensors revision, or other failure stages. ## Triage assessment - Impact level: P2 - Assessment status: supported - Impact basis: Two valid local Safetensors fixtures lost all parsed header observations when full hashing was interrupted after a real payload read; exact cancellation and descriptor cleanup remained correct. Successful inspections and the equivalent GGUF partial-result control delimit the failure. - Workaround status: available - Workaround basis: Header-only source-service inspections of both fixtures succeeded and returned the same declarations without entering payload hashing. The workaround requires the file to remain available and does not provide a full-file digest.

bsakaguchi closed this issue

2026-10-31 00:46:29 +00:00

bsakaguchi referenced this issue

2026-10-31 01:11:17 +00:00

[PR #4060] [CLOSED] fix(reference-source): keep a partial graph for symlinks into unenumerated directories #6027

bsakaguchi referenced this issue

2026-11-01 21:11:07 +00:00

[GH-ISSUE #5654] [Tracker] Open PRs that can't merge yet: conflicts and red CI (October 2026) #3425