morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #3323] [Feature] Inspect GGUF and safetensors model files without loading them #1194

New issue

Closed

opened 2026-10-15 18:31:53 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 18:31:53 +00:00

Owner

Originally created by @michaelkillgta on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3323

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

AI desktop apps, Electron apps and local LLM tools often ship model weights as GGUF (llama.cpp ecosystem) or safetensors (Hugging Face ecosystem) files. When an agent analyzing such an app reaches the model file, REA can't open it. Both inspect-artifact and extract-artifact reject it during target opening:

python3 - <<'PY'
import json, struct
h = json.dumps({"__metadata__": {"format": "pt"},
                "w": {"dtype": "F32", "shape": [2], "data_offsets": [0, 8]}}).encode()
open("/tmp/m.safetensors", "wb").write(struct.pack("<Q", len(h)) + h + b"\0" * 8)
PY
rea inspect-artifact /tmp/m.safetensors --format json

{"error":"Analysis failed","code":"target_unavailable","category":"unsupported_target","message":"Cannot open artifact: unsupported binary format.", ...}

A minimal GGUF v3 file (GGUF magic, version 3, zero tensors, one general.architecture key) fails the same way. The agent then has no evidence about which model the app uses, its architecture or quantization, or which chat template and tokenizer it embeds. Those facts often explain how an AI feature behaves.

Use cases

Identify the model behind an app feature: architecture, parameter count, quantization, context length.
Read the embedded chat template and special tokens to explain prompt formatting observed at runtime.
Compare two app versions' bundled models by digest and metadata (for example, a changed quantization or template).
Inventory model files found inside an extracted app bundle, alongside REA's existing artifact evidence.

Desired outcome and success criteria

A read-only operation (working name inspect_model_artifact / rea inspect-model-artifact <path>) that returns Evidence through the shared CLI/MCP model:

Common: artifact path, SHA-256, size, detected format and version, and limitations.
GGUF: version, tensor count, alignment, and metadata key/value pairs with their declared types. Scalar values are returned in full. Arrays are summarized as type and length with a bounded preview; the token vocabulary is not inlined by default. Selected keys are surfaced as observed facts, not interpretations: general.architecture, general.name, general.file_type, <arch>.context_length, tokenizer.ggml.model, tokenizer.chat_template. Each tensor is listed with name, ggml type, dimensions, offset and byte length.
safetensors: header length, __metadata__ as raw string pairs, and per-tensor dtype, shape and data offsets. Offsets are checked against the file size; gaps and overlaps are reported as findings.
Derived totals: parameter count and bytes by dtype. These are marked as derived from header declarations.
Paginated tensor lists (offset / limit), consistent with existing inventory tools.

Success: both formats open by content regardless of extension, malformed or truncated headers return typed file_format_rejected failures, and results stay within REA's output budgets.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Safety bounds:

Headers only. Tensor data is never read, loaded or executed. The parser reads the header region and file size only, so there is no model runtime, no Python and no network access.
Explicit caps on input header size (for example, the safetensors JSON header at 100 MB, the limit the format itself specifies, or lower), GGUF key/value and tensor counts, string and array lengths, and the total output. Exceeding a cap is a typed resource failure, not partial success.
Declared numbers are validated (counts, offsets, dimension products) with overflow-safe arithmetic before any allocation.
Out of scope: PyTorch pickle checkpoints (.pt, .bin, .ckpt), because a faithful reading needs unpickling semantics. They stay unsupported, with a remediation message.
Later: ONNX (protobuf graph and initializer inventory), as a separate follow-up once this contract is settled.

How it fits: this is format inspection, which the contribution scope lists as supported, with no execution, injection, mutation or protection recovery. It matches the roadmap admission criterion that "new capabilities must return useful evidence through shared CLI/MCP workflows, preserve observations and unknowns, and declare actual authority and lifecycle constraints." Unknown metadata types and unparsed regions would be preserved as unknowns, not dropped.

Verification available so far

I can suggest a sample or manual check

Verification details (optional)

Generated fixtures for both formats (valid; truncated header; oversized declared lengths; misaligned or overlapping offsets; unknown GGUF value type), built in tests without network access.
Optional cross-check against the reference readers (gguf Python package from llama.cpp, safetensors safe_open metadata) on small public models, as manual verification only, not a CI dependency.

Alternatives considered

Leave model files to external tools (gguf-dump, Python safetensors). That breaks the evidence chain and needs Python, which REA avoids for static paths.
Treat model files as opaque file nodes with a digest only. That loses the metadata agents actually need.

Possible direction

A pure TypeScript header reader under src/artifacts/ with a domain schema in src/domain/, content-based detection (GGUF magic; safetensors' little-endian u64 length followed by {), and contracts and CLI wiring following docs/tool-design.md. If the scope and output shape are agreed here, I'm happy to implement it as one focused PR with tests and docs, starting with GGUF and safetensors.

References

GGUF specification: https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
safetensors format: https://github.com/huggingface/safetensors#format
REA contribution scope: https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope
REA roadmap admission criteria: https://github.com/morluto/rea/blob/main/docs/roadmap.md#admission-criteria

Originally created by @michaelkillgta on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3323 ### Before submitting - [X] I searched the open and closed issues and pull requests for a related proposal. - [X] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [X] This request does not include credentials, private samples, or sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Current limitation AI desktop apps, Electron apps and local LLM tools often ship model weights as GGUF (llama.cpp ecosystem) or safetensors (Hugging Face ecosystem) files. When an agent analyzing such an app reaches the model file, REA can't open it. Both `inspect-artifact` and `extract-artifact` reject it during target opening: ```sh python3 - <<'PY' import json, struct h = json.dumps({"__metadata__": {"format": "pt"}, "w": {"dtype": "F32", "shape": [2], "data_offsets": [0, 8]}}).encode() open("/tmp/m.safetensors", "wb").write(struct.pack("<Q", len(h)) + h + b"\0" * 8) PY rea inspect-artifact /tmp/m.safetensors --format json ``` ```text {"error":"Analysis failed","code":"target_unavailable","category":"unsupported_target","message":"Cannot open artifact: unsupported binary format.", ...} ``` A minimal GGUF v3 file (`GGUF` magic, version 3, zero tensors, one `general.architecture` key) fails the same way. The agent then has no evidence about which model the app uses, its architecture or quantization, or which chat template and tokenizer it embeds. Those facts often explain how an AI feature behaves. ### Use cases - Identify the model behind an app feature: architecture, parameter count, quantization, context length. - Read the embedded chat template and special tokens to explain prompt formatting observed at runtime. - Compare two app versions' bundled models by digest and metadata (for example, a changed quantization or template). - Inventory model files found inside an extracted app bundle, alongside REA's existing artifact evidence. ### Desired outcome and success criteria A read-only operation (working name `inspect_model_artifact` / `rea inspect-model-artifact <path>`) that returns Evidence through the shared CLI/MCP model: - **Common:** artifact path, SHA-256, size, detected format and version, and limitations. - **GGUF:** version, tensor count, alignment, and metadata key/value pairs with their declared types. Scalar values are returned in full. Arrays are summarized as type and length with a bounded preview; the token vocabulary is not inlined by default. Selected keys are surfaced as observed facts, not interpretations: `general.architecture`, `general.name`, `general.file_type`, `<arch>.context_length`, `tokenizer.ggml.model`, `tokenizer.chat_template`. Each tensor is listed with name, ggml type, dimensions, offset and byte length. - **safetensors:** header length, `__metadata__` as raw string pairs, and per-tensor dtype, shape and data offsets. Offsets are checked against the file size; gaps and overlaps are reported as findings. - **Derived totals:** parameter count and bytes by dtype. These are marked as derived from header declarations. - Paginated tensor lists (`offset` / `limit`), consistent with existing inventory tools. Success: both formats open by content regardless of extension, malformed or truncated headers return typed `file_format_rejected` failures, and results stay within REA's output budgets. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) Safety bounds: - **Headers only.** Tensor data is never read, loaded or executed. The parser reads the header region and file size only, so there is no model runtime, no Python and no network access. - **Explicit caps** on input header size (for example, the safetensors JSON header at 100 MB, the limit the format itself specifies, or lower), GGUF key/value and tensor counts, string and array lengths, and the total output. Exceeding a cap is a typed resource failure, not partial success. - **Declared numbers are validated** (counts, offsets, dimension products) with overflow-safe arithmetic before any allocation. - **Out of scope:** PyTorch pickle checkpoints (`.pt`, `.bin`, `.ckpt`), because a faithful reading needs unpickling semantics. They stay unsupported, with a remediation message. - **Later:** ONNX (protobuf graph and initializer inventory), as a separate follow-up once this contract is settled. How it fits: this is format inspection, which the contribution scope lists as supported, with no execution, injection, mutation or protection recovery. It matches the roadmap admission criterion that "new capabilities must return useful evidence through shared CLI/MCP workflows, preserve observations and unknowns, and declare actual authority and lifecycle constraints." Unknown metadata types and unparsed regions would be preserved as unknowns, not dropped. ### Verification available so far I can suggest a sample or manual check ### Verification details (optional) - Generated fixtures for both formats (valid; truncated header; oversized declared lengths; misaligned or overlapping offsets; unknown GGUF value type), built in tests without network access. - Optional cross-check against the reference readers (`gguf` Python package from llama.cpp, `safetensors` `safe_open` metadata) on small public models, as manual verification only, not a CI dependency. ### Alternatives considered - Leave model files to external tools (`gguf-dump`, Python `safetensors`). That breaks the evidence chain and needs Python, which REA avoids for static paths. - Treat model files as opaque `file` nodes with a digest only. That loses the metadata agents actually need. ### Possible direction A pure TypeScript header reader under `src/artifacts/` with a domain schema in `src/domain/`, content-based detection (`GGUF` magic; safetensors' little-endian u64 length followed by `{`), and contracts and CLI wiring following `docs/tool-design.md`. If the scope and output shape are agreed here, I'm happy to implement it as one focused PR with tests and docs, starting with GGUF and safetensors. ### References - GGUF specification: https://github.com/ggml-org/ggml/blob/master/docs/gguf.md - safetensors format: https://github.com/huggingface/safetensors#format - REA contribution scope: https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope - REA roadmap admission criteria: https://github.com/morluto/rea/blob/main/docs/roadmap.md#admission-criteria

bsakaguchi commented

2026-10-15 18:31:54 +00:00

Author

Owner

@Y0UW1SH commented on GitHub (Oct 14, 2026):

I support this approach: detection by content, header-only reads, caps checked before allocation, typed failures, and derived totals labelled as derived. Apps that bundle on-device models also ship other formats. For example, the Electron app in #3538 bundles onnxruntime-web. The issue already lists ONNX under "Later". I would like to propose ONNX, TFLite and Core ML as the follow-up scope once this contract settles. Each format would report header and graph facts only:

ONNX (ModelProto protobuf): ir_version, opset_import (domain and version), producer name and version, model_version, and metadata_props. Also graph inputs and outputs with element type and shape (dim_value or dim_param), the node count, an op_type histogram per domain, and an initializer inventory (name, data type, dims, byte range). External-data references are reported as location, offset and length, and are never opened.
TFLite (FlatBuffer, TFL3 file identifier): schema version, description, and operator_codes (builtin or custom code, operator version). Per subgraph: tensors (name, shape, type, buffer index, whether quantization parameters are present), inputs and outputs, and an operator histogram. Also signature_defs, metadata entry names, and buffer sizes only, including the offset/size form used by models over 2 GB.
Core ML: for .mlmodel (the Model protobuf), specificationVersion, the model type, input and output feature descriptions, and the author, license, version and user-defined metadata. For .mlpackage, Manifest.json and its root model entry. The internals of a compiled .mlmodelc, beyond its JSON metadata, stay unknown.

The non-goals stay the same as here: no weights loaded into memory or dequantized, no inference, no format conversion, no network. Encrypted Core ML models are detected and reported, never decrypted.

Two points need settling first:

ONNX and .mlmodel are untagged protobuf, and both usually start with field 1 as a varint. "Opens by content" therefore needs a bounded top-level field walk that matches each schema's field numbers and wire types, with ambiguous results reported as unrecognized. REA has no protobuf or FlatBuffer reader today (source-derived, commit 1579b68).
.mlpackage and .mlmodelc are directories, while this issue's input is a single file.

Question for maintainers: once GGUF and safetensors land, should ONNX, TFLite and Core ML extend the same operation (working name inspect_model_artifact), or would you prefer a separate follow-up issue for them? I can open that issue if it is the better route.

Specs: onnx.proto, Core ML format protos, and the TFLite schema.fbs in tensorflow/tensorflow (revision to be pinned).

@Y0UW1SH commented on GitHub (Oct 14, 2026): I support this approach: detection by content, header-only reads, caps checked before allocation, typed failures, and derived totals labelled as derived. Apps that bundle on-device models also ship other formats. For example, the Electron app in #3538 bundles `onnxruntime-web`. The issue already lists ONNX under "Later". I would like to propose ONNX, TFLite and Core ML as the follow-up scope once this contract settles. Each format would report header and graph facts only: - **ONNX** (`ModelProto` protobuf): `ir_version`, `opset_import` (domain and version), producer name and version, `model_version`, and `metadata_props`. Also graph inputs and outputs with element type and shape (`dim_value` or `dim_param`), the node count, an `op_type` histogram per domain, and an initializer inventory (name, data type, dims, byte range). External-data references are reported as location, offset and length, and are never opened. - **TFLite** (FlatBuffer, `TFL3` file identifier): schema `version`, `description`, and `operator_codes` (builtin or custom code, operator version). Per subgraph: tensors (name, shape, type, buffer index, whether quantization parameters are present), inputs and outputs, and an operator histogram. Also `signature_defs`, `metadata` entry names, and buffer sizes only, including the offset/size form used by models over 2 GB. - **Core ML**: for `.mlmodel` (the `Model` protobuf), `specificationVersion`, the model type, input and output feature descriptions, and the author, license, version and user-defined metadata. For `.mlpackage`, `Manifest.json` and its root model entry. The internals of a compiled `.mlmodelc`, beyond its JSON metadata, stay `unknown`. The non-goals stay the same as here: no weights loaded into memory or dequantized, no inference, no format conversion, no network. Encrypted Core ML models are detected and reported, never decrypted. Two points need settling first: - ONNX and `.mlmodel` are untagged protobuf, and both usually start with field 1 as a varint. "Opens by content" therefore needs a bounded top-level field walk that matches each schema's field numbers and wire types, with ambiguous results reported as `unrecognized`. REA has no protobuf or FlatBuffer reader today (source-derived, commit 1579b68). - `.mlpackage` and `.mlmodelc` are directories, while this issue's input is a single file. Question for maintainers: once GGUF and safetensors land, should ONNX, TFLite and Core ML extend the same operation (working name `inspect_model_artifact`), or would you prefer a separate follow-up issue for them? I can open that issue if it is the better route. Specs: [onnx.proto](https://github.com/onnx/onnx/blob/main/onnx/onnx.proto), [Core ML format protos](https://github.com/apple/coremltools/tree/main/mlmodel/format), and the TFLite `schema.fbs` in [tensorflow/tensorflow](https://github.com/tensorflow/tensorflow) (revision to be pinned).

bsakaguchi referenced this issue

2026-10-17 21:09:19 +00:00

[PR #1194] [MERGED] fix(process): stop cancelled Swift helper builds before removing their files #2707

bsakaguchi referenced this issue

2026-10-17 21:09:36 +00:00

[PR #1221] [MERGED] fix(process): stop timed-out commands with the caller's stop signal #2730

bsakaguchi closed this issue

2026-10-31 00:30:03 +00:00

bsakaguchi referenced this issue

2026-10-31 00:30:34 +00:00

[GH-ISSUE #3537] [Tracker] Audit and subsystem index of active bug reports (October 2026) #1313

bsakaguchi referenced this issue