mirror of
https://github.com/morluto/rea.git
synced 2026-11-05 06:45:22 +00:00
[PR #5918] [MERGED] fix(model): admit safetensors header lengths that resemble pickle markers #7106
Closed
opened 2026-11-01 21:45:26 +00:00 by bsakaguchi
·
0 comments
No Branch/Tag specified
main
codex/procedure-reference-stacked
release-please--branches--main--components--rea-agents
release-please--branches--main--components--rea-agents--release-notes
codex/kuna-independent-retry
codex/byte-search-overlay-coverage
codex/open-failure-target-receipt
codex/kuna-xref-evidence
codex/ghidra-ambiguous-procedure-recovery
codex/function-source-bundle
codex/trace-function-integration-prototype
codex/snapshot-source-closure
codex/recorded-evidence-array-pages
codex/evidence-recovery-redesign
codex/trace-feature-address-scope
codex/call-discovery-scope
codex/ghidra-literal-candidate-evidence
codex/decompile-directory-independent-retry
codex/retry-provenance-base
codex/value-flow-capability-experiment
codex/ghidra-containing-instruction
cursor/readme-discord-button-ef9f
dependabot/github_actions/actions/setup-node-7.1.0
dependabot/github_actions/actions/upload-artifact-7.0.2
dependabot/npm_and_yarn/npm-minor-and-patch-d8d97ce6df
codex/kuna-1989
fix/8559-native-ui-fixture-ownership
fix/8555-go-verifier-mcp-ownership
codex/provider-capability-recovery
fix/8642-rizin-target-stdin
codex/ghidra-memory-effect-scope
codex/ghidra-annotation-entry-selector
codex/ghidra-readonly-recovery
codex/ghidra-regex-verifier-recovery
codex/evidence-discovery-view-guidance
fix/8545-bulk-append-overflow
fix/8428-navigation-refusal-alternatives
codex/address-context-recovery
codex/matching-member-limitations
codex/unknown-scope-contract
codex/evidence-view-guidance
feat/8373-native-trace-views
codex/ghidra-snapshot-verifier-binding
codex/elf-export-visibility
codex/evidence-view-recovery
codex/native-reference-cli-help
codex/provider-candidate-observation-scope
codex/kuna-session-verification
codex/ghidra-switch-lifetime-guidance
codex/pseudocode-paging-discovery
codex/clarify-evidence-view-support
codex/kuna-xrefs-selector-recovery
codex/context-capacity-recovery
codex/evidence-closure-large-links
codex/same-function-evidence-identity
codex/ghidra-verifier-inspected-jdk
codex/audit-guided-workflows
codex/crash-prompt-retained-dossiers
codex/analysis-view-error-recovery
codex/kuna-verifier-target-selection
codex/ghidra-selected-project-call-coverage
codex/docs-native-analysis-scope
codex/ghidra-java-socket-boundary
codex/ghidra-retained-snapshot-verifier
codex/ghidra-search-inventory-facts
fix/8284-zero-argument-parameter-bindings
fix/8198-ui-action-handler-offset
fix/8257-elf-entry-label-identity
fix/8304-deferred-output-schema-projection
fix/8277-call-path-computed-target-uncertainty
fix/pr-8089-conflicts
fix/ci-static-duplicate-export
dependabot/npm_and_yarn/lint-staged-17.6.0
dependabot/github_actions/actions/download-artifact-8.0.2
dependabot/npm_and_yarn/write-file-atomic-8.0.0
dependabot/npm_and_yarn/canonicalize-5.1.0
dependabot/github_actions/actions/cache/restore-6.1.0
dependabot/npm_and_yarn/toon-format/toon-4.4.0
dependabot/github_actions/actions/deploy-pages-5.0.1
codex/review-fixes-pr-2403
codex/review-fixes-pr-2400
codex/process-terminal-width
codex/test-brittleness-cleanup
verify/7167-browser-connect-macos
fix/7167-browser-connect
feat/native-matching-allowance-setting
feat/browser-scenario-dry-run
test/managed-caller-parameters-20261024
codex/parser-fortythird-boundaries
fix/6926-pinvoke-platform-decorations-20261023
fix/6889-har-body-encoding-20261023
fix/6888-sqlite-schema-budget-20261023
fix/dependency-security-gates-6708
cursor/evidence-overwrite-rebase-b4ac
feat/4057-verify-native-exports
fix/windows-acl-observer-overhead-20261022
cursor/erlang-beam-restack-4476
cursor/pe-certificate-review-bf32
fix/6116-javascript-application-fs-errors
fix/6061-unmapped-runtime-script-unknown
feat/browser-scenario-summary
fix/pe-certificates
codex/verify-3407
cursor/rebase-5699
cursor/rebase-5352
cursor/rebase-5600
cursor/fix-cli-schema-omission-10a8
codex/audit-hopper-shutdown-reasons-20261021
fix/demangle-swift-omitted-record
fix/nib-boolean-value-types
cursor/cancellation-partial-observation-a5ac
fix/ib-object-limit-reasons
fix/electron-fuse-limit-advice
fix/kuna-doctor-dangling-cache
fix/fixed-budget-refusal-advice
fix/js-ipc-listener-dedup
fix/js-return-member-provenance
fix/dmg-integrity-retry-advice
fix/ida-doctor-readiness
fix/kuna-version-probe-output-drain
fix/legacy-swift-runtime-name-matching
fix/dylib-lazy-load-commands
fix/browser-scenario-startup-cancel
fix/swift-nested-punycode-classification
fix/compare-unselected-scopes
fix/objc-method-budget-coverage
fix/dmg-mounted-root-observations
audit/ci-tests-20261019-round2
release/6.7.0
fix/wabt-cleanup-retry-resolution
fix/electron-fuse-elf-pe-architecture
fix/native-plist-descriptor-ownership
fix/kuna-caller-cleanup-ownership
experiment/atomic-codemode-20261016
review/first-contributor-3776
fix/call-resolution-full-read-cycles
fix/member-mutation-selection-repeat
experiment/atomic-re-tools-20261015
perf/member-mutation-lazy-origins
perf/static-network-mutation-index
fix/hopper-startup-lifecycle
audit/3050-windows-proof-20261015
feat/windows-window-capture-2183
audit/3049-windows-proof-20261015
audit/3735-macos-fixtures-20261015
fix/batch-decompile-null-pseudocode
fix/kuna-inventory-honors-jobs
experiment/browser-codemode-tool-design-20261015
fix/hopper-clean-runtime-argv
codex/review-2049-27f1b367
review-fixes/pr-2787-routing
fix/3360-scoped-this-emitters
fix/3367-replaced-global-effects
fix/3838-compact-any-value-items
test/hopper-download-probe-20261015
codex/windows-runtime-creation-red-green-20261015
codex/windows-readonly-root-red-green-20261015
codex/windows-root-share-red-green-20261015
codex/exp-1685-win-mcp-startup
codex/bug-sweep-20261015
investigate/2190-windows-package-phases
investigate/3543-macos-cli-deadline
perf/3281-reverse-tools-feedback-quality
fix/3406-hopper-doctor-fixture
fix/3341-wrapper-require-test-context
fix/3301-owned-uninstall-fixtures
fix/3164-compiler-input-ownership
fix/pcap-cleanup-stdout-newline-20261014
perf/interface-builder-summary
codex/review-fixes-pr-1977
fix/ghidra-effective-jvm-platform
fix/docs-turkish-rust-link-20261013
fix/main-maintenance-ci-20261010
fix/recover-binary-session-process-effects
codex/review-fixes-pr-2457
codex/ci-required-cancellation-gate
fix/main-format-baseline-20261013
fix/setup-preserve-entry-fields
cursor/restore-main-checks-1ffc
fix/ghidra-bridge-hashmap-import
codex/shared-ci-jadx-json-default-20261013
codex/shared-ci-ghidra-test-groups-20261013
codex/shared-ci-java-fixture-format-20261013
codex/fix-intrinsic-invocation-boundaries
test/release-preflight-reusable-jobs
test/js-parameter-provenance-expectation
fix/linux-hopper-verification-current
test/2419-ghidra-elf-probe-size
followup/pr-1544-native-ida
codex/fold-pr-2230-contribution
docs/readme-responsible-use
fix/issue-2064
fix/issue-2050
fix/issue-2053
fix/issue-1968
fix/issue-1945
experiment/codemode-bridge-20261011
fix/tool-effects-negative-wording-1904-20261012
fix/1842-electron-interop-provenance
feat/recover-javascript-multi-input
refactor/website-i18n-prep
fix/nativeaot-pe-virtual-extent
fix/browser-verifier-ci-hang
fix/pe-resources-input-selection
docs/beginner-friendly-guides
docs/beginner-setup-guide
fix/javascript-analysis-worker-1563
fix/1649-ghidra-duplicate-names
fix/javascript-export-keys-integration-1369
test/maintenance-boundary-contracts-20261011
review/javascript-export-keys-1369
review/process-live-progress-1483
test/javascript-1495-growth-ratio
fix/ci-escape-timing
docs/normalized-terminal-geometry-1450
release/6.3.0
archive/release-6.3.0-pr1381
release/6.2.0
fix/electron-preload-dynamic-override
fix/javascript-self-referential-bundler-edges
release/6.1.0
cursor/grok-client-cherry-pick-0b56
feat/signature-posture
feat/dyld-shared-cache
feat/artifact-integrity-policy
feat/pkg-inventory
release/6.0.0
chore/sync-release-5.0.0
release/5.0.0
cursor/cloud-agent-node-787e
fix/javascript-xhr-open-endpoints
review/pr-711-realpath-fallback-wording
cleanup/replay-confinement
codex/test-suite-performance
codex/issue-foundations
codex/address-open-issues-2026-07-20
agent/javascript-vite-rollup-esbuild-adapters
agent/process-cleanup
agent/triage-integration
agent/unknown-commands-registry
feat/mcp-identity-metadata
morlvps/issue_cleanup
codex/installer-redesign
morl/roadmap-investigation-core
morl/roadmap-investigation-docs
release-please--branches--main--components--rea
rea-agents-6.8.0
rea-agents-6.7.0
rea-agents-6.6.0
experiment/atomic-codemode-wait-runtime-20261016
rea-agents-6.5.0
rea-agents-6.4.0
rea-agents-6.3.0
rea-agents-6.2.0
rea-agents-6.1.0
rea-agents-6.0.0
rea-agents-5.0.0
rea-agents-4.1.0
rea-agents-4.0.1
rea-agents-4.0.0
rea-agents-3.2.1
rea-agents-3.2.0
rea-agents-3.1.0
rea-agents-3.0.0
rea-agents-2.7.0
rea-agents-2.6.0
rea-agents-2.5.1
rea-agents-2.5.0
rea-agents-2.4.0
rea-agents-2.3.0
rea-agents-2.2.0
rea-agents-2.1.0
rea-agents-2.0.0
rea-agents-1.7.0
rea-agents-1.6.0
rea-agents-1.5.0
rea-agents-1.4.0
rea-agents-1.3.0
rea-agents-1.2.0
rea-agents-1.1.0
rea-agents-1.0.0
rea-agents-0.5.0
rea-agents-0.4.0
rea-agents-0.3.0
rea-0.2.1
rea-0.2.0
Labels
Clear labels
area: bridge
area: bridge
area: build
area: build
area: build
area: cli
area: domain
area: evidence
area: mcp
area: native
area: providers
area: runtime
area: server
area: setup
area: tests
bug
design
documentation
enhancement
enhancement
enhancement
experiment
help wanted
i18n
invalid
P0: critical
P1: high
P2: medium
P2: medium
P2: medium
P3: low
performance
provider-request
pull-request
Mirrored from GitHub Pull Request
question
roadmap
triage
No labels
area: bridge
area: bridge
area: build
area: build
area: build
area: cli
area: domain
area: evidence
area: mcp
area: native
area: providers
area: runtime
area: server
area: setup
area: tests
bug
design
documentation
enhancement
enhancement
enhancement
experiment
help wanted
i18n
invalid
P0: critical
P1: high
P2: medium
P2: medium
P2: medium
P3: low
performance
provider-request
pull-request
question
roadmap
triage
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
morluto/rea#7106
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
📋 Pull Request Information
Original PR: https://github.com/morluto/rea/pull/5918
Author: @amyanger
Created: 10/21/2026
Status: ✅ Merged
Merged: 10/22/2026
Merged by: @morluto
Base:
main← Head:fix/5895-safetensors-pickle-sniff📝 Commits (1)
9ac6d48fix(model): admit safetensors header lengths that resemble pickle markers📊 Changes
4 files changed (+94 additions, -9 deletions)
View changed files
📝
src/model/ModelArtifactProvider.ts(+2 -8)📝
src/model/ModelFormat.ts(+7 -0)📝
src/model/SafetensorsHeader.ts(+11 -1)📝
tests/boundary/model/modelArtifact.test.ts(+74 -0)📄 Description
Summary
Fixes #5895.
inspect_model_artifactrefused valid safetensors files whose JSON header length is 640, 896, 1152, 1408, 66176 and so on. Those lengths encode as80 02..05 ...in the 8-byte little-endian prefix, which the provider read as a pickle PROTO marker. The CLI exited 1 withunsupported_targetand asked for safetensors, although the input already was safetensors. These files now parse. Genuine pickles are still refused as unsupported and are never deserialized.Problem and expected behavior
Smallest trigger: a safetensors file with one
F32tensor whose header is padded to 640 bytes. Its prefix is80 02 00 00 00 00 00 00followed by{. The provider checked only the first two bytes (0x80, then 2 to 5) and threw the pickle refusal before the safetensors parser ran.The safetensors format defines an 8-byte unsigned little-endian length followed by UTF-8 JSON that begins with
{. Nothing in it excludes these lengths, so a file with that structure should be parsed.Change and scope
The pickle decision moved from
ModelArtifactProvider.#readintoreadSafetensorsHeader, the parser that already owns the length prefix and the{check. For a pickle-like prefix it:ModelHeaderReader, which is hashed, abort-checked and loops on short reads;{and the declared length is withinMODEL_HEADER_BYTES(100,000,000), the safetensors header ceiling the parser already enforces.A file that passes continues through the normal parser. So a truncated safetensors file with a colliding length now gets the existing
file_format_rejectederror for an out-of-file header instead of the pickle diagnosis. The refusal message moved into a sharedrejectPickle()inModelFormat.ts, used by the ZIP check and the parser. Its text is unchanged.Precedent: the parser already rejects any non-
{byte 8 and any length beyondMODEL_HEADER_BYTES. The tiebreak uses those same facts rather than a new heuristic.Not included: the ZIP magic
50 4B 03 04collides the same way with exactly one safetensors header length, 67,324,752. #5895 does not report it, so I left it out to keep this PR to one outcome. It can be a follow-up issue.Scope decision: not applicable. This is a fix inside the existing model-artifact boundary.
Contract and boundary impact
src/model/SafetensorsHeader.ts(readSafetensorsHeader), at format detection.file_format_rejectedinstead ofunsupported_target.Expected effects:
Evidence and regression coverage
tests/boundary/model/modelArtifact.test.ts:weighttensor;{}, a protocol 4 FRAME pickle, the legacytorch.savemagic, the protocol 2 pickle of"x{"(brace at byte 8, inadmissible length) and an in-file length with no brace;BinaryTargetErrorfile_format_rejected, not pickle.d672cc239with the issue's nine-case script: lengths 640, 896, 1152 and 1408 exit 1 withunsupported_targetandPyTorch pickle/ZIP model deserialization is unsupported; supply GGUF or safetensors instead. Lengths 632, 648, 904, 1160 and 1416 parse.src/model): 6 failed, 74 passed. The failures are the five admission cases and the truncation case. The refusal cases pass on base, as they should."exit":0,"format":"safetensors". Real Python pickles at protocols 2, 3, 4 and 5 andpickle.dumps('x{', 2)printunsupported_target.< 9to< 5: 1 fails (the 6-byte{}pickle);"x{");The
>to>=mutant at the ceiling is equivalent. 100,000,000 is0x05F5E100, whose low byte is0x00, so no pickle-like prefix can equal it.unsupported. It needs byte 3 at most 5, bytes 4 to 7 zero and byte 8{, for examplepickle.dumps('\x00{', 2)orpickle.dumps(b'\x00\x00\x00\x00{', 3). Those bytes are also a valid safetensors prefix, so no prefix rule can classify both. Nothing is deserialized either way. A reviewer ran about 810,000 generated pickles at protocols 2 to 5 (state-dict shapes, nested objects, strings full of{, the torch magic) through the rule. Only this crafted class was routed to safetensors. Protocol 4 and 5 cannot be: byte 8 is a high byte of the FRAME length.For evidence-bearing changes:
Validation performed
Development base:
d672cc23998b6cbb21c485f118595ebfb82fed33, the newest green upstreammainat the time; every base comparison below used it. Before publishing, the branch was rebased onto the newer greenmainc76edbebbe1e25c2bf821ec04d08dc851e37626c, which changes nothing undersrc/modelortests/boundary/model. After the rebase: the focused command below passed 90 tests,npm run check:fastpassed andnpm run test:changed -- --base c76edbebbe1e25c2bf821ec04d08dc851e37626cpassed 3 tests.npm run test:focused -- tests/boundary/model/modelArtifact.test.ts tests/acceptance/modelArtifactCli.test.ts tests/boundary/model/modelArtifactPartialCancellation.test.ts tests/boundary/mcp/modelArtifactPartialCancellation.test.ts: 4 files, 90 tests passed.npm run check:fast: passed.npm run test:changed -- --base d672cc23998b6cbb21c485f118595ebfb82fed33: 2 files, 3 tests passed. It selects source tests only; the built model boundary tests ran above.npm run check: passed, including module boundaries and architecture guards. Last run before the test-only addition of the 6-byte pickle case;check:fastand the focused tests ran after it.npx oxfmt --check src/model tests/boundary/model: passed.npm run build:cachedand then the issue's reproduction script throughscripts/rea.mjs inspect-model-artifact: all nine lengths parse as safetensors.'x{'): allunsupported_target.npm run docs:checkandnpm run verify:test-discovery: not run. No contract, generated metadata or new test files.Review:
code-review(high), round 1: the first version used a rawfile.readside probe that skipped short-read looping and abort checks and duplicated the length decode. Fixed by moving the decision into the parser, through the reader."x{"pickle was misrouted. Resolved with the header-ceiling condition and a test. The human owner chose that tiebreak over two alternatives.first[1] ?? 0, whichnoUncheckedIndexedAccessrequires;take(4)first;docs/model-artifacts.mdonly says pickle deserialization is unsupported, which still holds.Compatibility, safety, and release
Author readiness
Human owner: @amyanger
Before merge
type(scope): outcome.🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.