morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8097] [Feature] Inspect ONNX Runtime .ort models (ORTM FlatBuffers) in inspect_model_artifact #4670

New issue

Open

opened 2026-10-31 00:52:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-31 00:52:15 +00:00

Owner

Originally created by @zdmilot on GitHub (Oct 30, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8097

Before submitting

I searched the open and closed issues and pull requests for a related proposal.
I understand the contribution scope must be reviewed before integration; open questions are welcome here.
This request does not include credentials, private samples, or sensitive target data.

Area

Evidence, artifacts, or reconstruction

Current limitation

inspect_model_artifact covers GGUF v2/v3 and safetensors. It has no reader for the ONNX Runtime .ort format — ORT's own FlatBuffers serialization, identified by the ORTM file identifier at bytes 4..8. This is a different container from the .onnx protobuf graph tracked in #3637.

The distinction matters in practice because .ort is ORT's deployment format. Shipped desktop and mobile products that embed ONNX Runtime commonly ship .ort rather than .onnx, because the conversion is the documented step for size- and load-time-constrained targets. So when an analyst meets an ONNX Runtime model inside a real product, .ort is a likely encounter, and inspect_model_artifact currently dead-ends on it.

Observed on a commercial Windows barcode-reader SDK bundle (Dynamsoft, nine neural model files). Request and result:

inspect_model_artifact { "path": "<bundle>/Models/DataMatrixQRCodeDeblur.data" }
-> error.code "target_unavailable", category "unsupported_target"
   message:  "Cannot open artifact: File does not contain a GGUF or safetensors header."
   details:  { "reason": "File does not contain a GGUF or safetensors header",
               "constraint": "file_format_rejected" }

That refusal is correct for the documented contract — src/contracts/model/modelToolContracts.ts scopes the tool to "local GGUF v2/v3 or safetensors declarations", and src/model/ModelFormat.ts carries only rejectModel and rejectPickle paths. This is a request to widen coverage, not a bug report.

Use cases

Explaining an on-device feature in a shipped application when the model is .ort: which inputs it takes, what the preprocessing contract is, which operators and domains it requires.
Recovering the input/output tensor contract without installing or running ONNX Runtime, which is the same read-only, no-execution posture #3323 established for GGUF and safetensors.
Relating a model to the native code that drives it. In the observed bundle the loader is reachable statically and names its model classes, so a model-side inventory would compose with existing native analysis.

No frequency measurements. One observed bundle; I have not surveyed how common .ort is across products.

Desired outcome and success criteria

For a selected .ort file, return observed declarations with Evidence and explicit unknowns, without loading or executing the model:

Format identity: ORTM identifier and the ORT format version string.
Graph inputs and outputs: names, element types, and shapes with symbolic dimensions preserved as symbols rather than collapsed to a number or dropped. In the observed models these carry meaning (['batch', 3, 'height', 'width'], [1, 1, 16, 'width']), so flattening them would lose the preprocessing contract.
Operator inventory: op type, domain, and since-version. These are observable as readable strings in the files, for example :ConvInteger:10, :MaxPool:12, com.microsoft:QuickGelu:1. The com.microsoft entry is the kind of fact worth surfacing, since it marks a contrib operator and implies a runtime requirement.
Initializer and tensor inventory: names, types, shapes, and byte extents, consistent with how the GGUF and safetensors path reports tensors.
Coverage and limitations in the existing style: partial when tables are unreadable, and unknown rather than absent for unresolved types.

Success criterion: for an .ort model, an analyst gets the I/O contract and operator requirements without running a model runtime, and complete Evidence is retained for inspect_analysis_view paging the way the GGUF and safetensors summary already does.

Expected interaction with targets

Read existing files or captured data

Target and scope details (optional)

Non-goals for this request: no inference, no tensor-payload interpretation, no accuracy claims, and no protobuf .onnx support, which is #3637.

Scope question for maintainers, flagged explicitly rather than assumed. In the observed bundle the .ort files are not loose on disk. Each is wrapped in a small vendor container (a 24-byte NUL-padded magic such as DM_BARCODE_DEBLUR, a uint64 header length, a JSON table mapping member -> [offset, length], then the members) and the whole file is byte-obfuscated with a key derived from the file length. Reaching the .ort bytes therefore required undoing a vendor protection.

I am not requesting that REA deobfuscate vendor model containers. Per the contribution scope, protection recovery needs a maintainer scope decision, and I have not sought one. What I would like is the .ort reader itself, which is pure read with no protection recovery. Whether REA should ever handle wrapped or obfuscated model containers is a separate question I am raising, not proposing. If the answer is no, a plain .ort reader remains useful, because unwrapped .ort files ship in plenty of applications.

A related framing already exists in #3714 (recursive inspection and candidate analyzer routing), which would cover "this blob is a container, here is what is inside" more generally.

Verification available so far

I have a prototype or observed results

Verification details (optional)

I extracted the .ort members from the observed bundle and loaded all nine in onnxruntime 1.27.0 using CPUExecutionProvider. All nine constructed an InferenceSession, and four ran inference on random input of the declared shape, producing finite outputs:

OneDv9m4s.ort                            in  [('images', ['batch', 3, 'height', 'width'])]
                                         out (1, 500, 9)               finite
ean13.ort                                in  [('input', [1, 1, 16, 256])]
                                         out (1, 128, 5) x2            finite
model_sq_normalized_X2_6B16G32F_40w.ort  in  [('input', ['batch_size', 3, 'height', 'width'])]
                                         out (1, 3, 128, 128)          finite   (2x upscale)
sq_rnx2_dm+qr_6b16g32f_100w.ort          in  [('input', ['batch_size', 3, 'height', 'width'])]
                                         out (1, 3, 128, 128)          finite

This establishes that the extracted members are well-formed ORT models and that the declarations a reader would report are real rather than inferred. It establishes nothing about REA, which has no .ort path today.

One caveat worth recording for whoever implements this: onnxruntime 1.27 accepted these models, while the bundle's own runtime is ORT 1.14 (the shipped DLL contains the string onnxruntime/blob/rel-1.14.0/docs/ORT_Format_Update_). I did not test the ORT format version floor, so I cannot say which ortModelVersion values a reader must tolerate. Treat cross-version acceptance as unverified.

I cannot attach the vendor's model files; the bundle is a licensed third-party SDK and not mine to redistribute. A maintainer-side sample can be produced with ORT's own converter from any public .onnx model, which yields a plain .ort with no vendor container.

Alternatives considered

inspect_model_artifact — refuses by documented scope, as shown above.
inspect_wasm_artifact, inspect_managed_artifact — wrong artifact families.
inspect_binary_layout, read_bytes, search_bytes, inspect_byte_entropy — these worked, and were how I confirmed the container, but they report bytes rather than model declarations. Recovering the I/O contract that way means hand-walking FlatBuffers vtables, which is the per-caller reinterpretation the tool-design guidance avoids by giving an interpretation one owner.
Running onnxruntime outside REA — what I did to verify, but it executes a model runtime and produces no Evidence, so it is not a substitute for a REA reader.

Possible direction

A proposal, not a requirement. A FlatBuffers reader over ORT's published ort.fbs schema, parallel to the existing GGUF and safetensors readers: format detection on the ORTM identifier, then tables for graph inputs and outputs, nodes, and initializers. Reusing the existing model header and report budgets and the ModelFormatFailure kinds would keep refusals typed the same way. Preserving symbolic dimensions as symbols is the main representational choice I would flag, since collapsing them loses the preprocessing contract that makes the result useful.

References

#3323 (closed) — established the read-only GGUF and safetensors contract this would extend, and named ONNX as a follow-up.
#3637 (open) — ONNX protobuf graph topology, plus TFLite, CoreML, and TensorRT. Related, but a different serialization: .ort is FlatBuffers rather than protobuf, and is not mentioned there.
#3714 (open) — universal artifact discovery and analyzer routing; the natural home for container detection.
ONNX Runtime ORT format documentation and the upstream ort.fbs schema.
Source scoping the current refusal: src/contracts/model/modelToolContracts.ts and src/model/ModelFormat.ts, at 317bf3fe2.

Observed with REA 6.8.0 (rea-agents npm package), Node 24.22.2, Windows 11 10.0.26100 x64, MCP interface.

Originally created by @zdmilot on GitHub (Oct 30, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8097 ### Before submitting - [x] I searched the open and closed issues and pull requests for a related proposal. - [x] I understand the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope) must be reviewed before integration; open questions are welcome here. - [x] This request does not include credentials, private samples, or sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Current limitation `inspect_model_artifact` covers GGUF v2/v3 and safetensors. It has no reader for the **ONNX Runtime `.ort` format** — ORT's own FlatBuffers serialization, identified by the `ORTM` file identifier at bytes 4..8. This is a different container from the `.onnx` protobuf graph tracked in #3637. The distinction matters in practice because `.ort` is ORT's *deployment* format. Shipped desktop and mobile products that embed ONNX Runtime commonly ship `.ort` rather than `.onnx`, because the conversion is the documented step for size- and load-time-constrained targets. So when an analyst meets an ONNX Runtime model inside a real product, `.ort` is a likely encounter, and `inspect_model_artifact` currently dead-ends on it. Observed on a commercial Windows barcode-reader SDK bundle (Dynamsoft, nine neural model files). Request and result: ``` inspect_model_artifact { "path": "<bundle>/Models/DataMatrixQRCodeDeblur.data" } -> error.code "target_unavailable", category "unsupported_target" message: "Cannot open artifact: File does not contain a GGUF or safetensors header." details: { "reason": "File does not contain a GGUF or safetensors header", "constraint": "file_format_rejected" } ``` That refusal is **correct for the documented contract** — `src/contracts/model/modelToolContracts.ts` scopes the tool to "local GGUF v2/v3 or safetensors declarations", and `src/model/ModelFormat.ts` carries only `rejectModel` and `rejectPickle` paths. This is a request to widen coverage, not a bug report. ### Use cases - Explaining an on-device feature in a shipped application when the model is `.ort`: which inputs it takes, what the preprocessing contract is, which operators and domains it requires. - Recovering the input/output tensor contract without installing or running ONNX Runtime, which is the same read-only, no-execution posture #3323 established for GGUF and safetensors. - Relating a model to the native code that drives it. In the observed bundle the loader is reachable statically and names its model classes, so a model-side inventory would compose with existing native analysis. No frequency measurements. One observed bundle; I have not surveyed how common `.ort` is across products. ### Desired outcome and success criteria For a selected `.ort` file, return observed declarations with Evidence and explicit unknowns, without loading or executing the model: - Format identity: `ORTM` identifier and the ORT format version string. - Graph inputs and outputs: names, element types, and shapes with symbolic dimensions preserved as symbols rather than collapsed to a number or dropped. In the observed models these carry meaning (`['batch', 3, 'height', 'width']`, `[1, 1, 16, 'width']`), so flattening them would lose the preprocessing contract. - Operator inventory: op type, domain, and since-version. These are observable as readable strings in the files, for example `:ConvInteger:10`, `:MaxPool:12`, `com.microsoft:QuickGelu:1`. The `com.microsoft` entry is the kind of fact worth surfacing, since it marks a contrib operator and implies a runtime requirement. - Initializer and tensor inventory: names, types, shapes, and byte extents, consistent with how the GGUF and safetensors path reports tensors. - Coverage and limitations in the existing style: partial when tables are unreadable, and unknown rather than absent for unresolved types. Success criterion: for an `.ort` model, an analyst gets the I/O contract and operator requirements without running a model runtime, and complete Evidence is retained for `inspect_analysis_view` paging the way the GGUF and safetensors summary already does. ### Expected interaction with targets Read existing files or captured data ### Target and scope details (optional) **Non-goals** for this request: no inference, no tensor-payload interpretation, no accuracy claims, and no protobuf `.onnx` support, which is #3637. **Scope question for maintainers, flagged explicitly rather than assumed.** In the observed bundle the `.ort` files are not loose on disk. Each is wrapped in a small vendor container (a 24-byte NUL-padded magic such as `DM_BARCODE_DEBLUR`, a `uint64` header length, a JSON table mapping `member -> [offset, length]`, then the members) and the whole file is byte-obfuscated with a key derived from the file length. Reaching the `.ort` bytes therefore required undoing a vendor protection. I am **not** requesting that REA deobfuscate vendor model containers. Per the [contribution scope](https://github.com/morluto/rea/blob/main/CONTRIBUTING.md#contribution-scope), protection recovery needs a maintainer scope decision, and I have not sought one. What I would like is the `.ort` reader itself, which is pure read with no protection recovery. Whether REA should ever handle wrapped or obfuscated model containers is a separate question I am raising, not proposing. If the answer is no, a plain `.ort` reader remains useful, because unwrapped `.ort` files ship in plenty of applications. A related framing already exists in #3714 (recursive inspection and candidate analyzer routing), which would cover "this blob is a container, here is what is inside" more generally. ### Verification available so far I have a prototype or observed results ### Verification details (optional) I extracted the `.ort` members from the observed bundle and loaded all nine in `onnxruntime` 1.27.0 using `CPUExecutionProvider`. All nine constructed an `InferenceSession`, and four ran inference on random input of the declared shape, producing finite outputs: ``` OneDv9m4s.ort in [('images', ['batch', 3, 'height', 'width'])] out (1, 500, 9) finite ean13.ort in [('input', [1, 1, 16, 256])] out (1, 128, 5) x2 finite model_sq_normalized_X2_6B16G32F_40w.ort in [('input', ['batch_size', 3, 'height', 'width'])] out (1, 3, 128, 128) finite (2x upscale) sq_rnx2_dm+qr_6b16g32f_100w.ort in [('input', ['batch_size', 3, 'height', 'width'])] out (1, 3, 128, 128) finite ``` This establishes that the extracted members are well-formed ORT models and that the declarations a reader would report are real rather than inferred. It establishes nothing about REA, which has no `.ort` path today. One caveat worth recording for whoever implements this: `onnxruntime` 1.27 accepted these models, while the bundle's own runtime is ORT 1.14 (the shipped DLL contains the string `onnxruntime/blob/rel-1.14.0/docs/ORT_Format_Update_`). I did not test the ORT format version floor, so I cannot say which `ortModelVersion` values a reader must tolerate. Treat cross-version acceptance as unverified. I cannot attach the vendor's model files; the bundle is a licensed third-party SDK and not mine to redistribute. A maintainer-side sample can be produced with ORT's own converter from any public `.onnx` model, which yields a plain `.ort` with no vendor container. ### Alternatives considered - `inspect_model_artifact` — refuses by documented scope, as shown above. - `inspect_wasm_artifact`, `inspect_managed_artifact` — wrong artifact families. - `inspect_binary_layout`, `read_bytes`, `search_bytes`, `inspect_byte_entropy` — these worked, and were how I confirmed the container, but they report bytes rather than model declarations. Recovering the I/O contract that way means hand-walking FlatBuffers vtables, which is the per-caller reinterpretation the tool-design guidance avoids by giving an interpretation one owner. - Running `onnxruntime` outside REA — what I did to verify, but it executes a model runtime and produces no Evidence, so it is not a substitute for a REA reader. ### Possible direction A proposal, not a requirement. A FlatBuffers reader over ORT's published `ort.fbs` schema, parallel to the existing GGUF and safetensors readers: format detection on the `ORTM` identifier, then tables for graph inputs and outputs, nodes, and initializers. Reusing the existing model header and report budgets and the `ModelFormatFailure` kinds would keep refusals typed the same way. Preserving symbolic dimensions as symbols is the main representational choice I would flag, since collapsing them loses the preprocessing contract that makes the result useful. ### References - #3323 (closed) — established the read-only GGUF and safetensors contract this would extend, and named ONNX as a follow-up. - #3637 (open) — ONNX protobuf graph topology, plus TFLite, CoreML, and TensorRT. Related, but a different serialization: `.ort` is FlatBuffers rather than protobuf, and is not mentioned there. - #3714 (open) — universal artifact discovery and analyzer routing; the natural home for container detection. - ONNX Runtime ORT format documentation and the upstream `ort.fbs` schema. - Source scoping the current refusal: `src/contracts/model/modelToolContracts.ts` and `src/model/ModelFormat.ts`, at `317bf3fe2`. Observed with REA 6.8.0 (`rea-agents` npm package), Node 24.22.2, Windows 11 10.0.26100 x64, MCP interface.

bsakaguchi referenced this issue

2026-10-31 01:13:48 +00:00

[PR #4670] [CLOSED] fix(adb): report unknown requested permissions when package details are unavailable #6354

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#4670

No description provided.

Rows
Columns