[GH-ISSUE #3738] [Feature] Polkadot/Substrate reverse engineering: SCALE metadata, runtime Wasm, pallets, extrinsics and runtime-upgrade diffs #1447

New issue

Open

opened 2026-10-16 21:05:31 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:31 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3738

Problem Statement & Context

Polkadot and Substrate-based blockchains employ a unique runtime architecture: the entire blockchain state transition logic is compiled into a WebAssembly binary (runtime.wasm) stored directly within the on-chain storage. When a blockchain upgrade occurs, the runtime Wasm is overwritten on-chain (forkless runtime upgrade), fundamentally altering pallets, storage structures, extrinsics, and dispatch calls.

Reverse engineering a Substrate chain, node snapshot, or transaction requires decoding:

SCALE Codec (Simple Concatenated Aggregate Little-Endian): Compact binary serialization format used across Substrate for types, storage entries, and network packets.
SCALE Runtime Metadata (V14 & V15): Self-describing type registry (PortableRegistry) and pallet manifests embedded in the runtime. Defines every pallet, call function (Call), storage entry, event, custom error, and constant.
Runtime Wasm Blob (:code): Compressed or uncompressed WebAssembly module implementing runtime API entrypoints (Core_version, Metadata_metadata, BlockBuilder_apply_extrinsic).
Pallet Architecture & Extrinsics: Modularity decomposed into individual pallets (System, Balances, Staking, Governance). Extrinsics (transactions) encoded as pallet index + call index + arguments.
Runtime Upgrade Diffing: Comparing Metadata V14/V15 and Wasm binaries between Block N and Block N+1 to determine what pallets were added, storage layouts migrated, or call signatures modified.

REA needs a dedicated Substrate engine capable of parsing SCALE metadata, disassembling runtime Wasm entrypoints, decoding extrinsics, and performing semantic diffs across runtime upgrades.

Proposed Architecture

flowchart TD
    A["Substrate Artifact (Metadata Hex / runtime.wasm / Block Extrinsic)"] --> B["Substrate / Polkadot Semantic Engine"]
    B --> C["SCALE Codec Deserializer"]
    B --> D["Metadata V14 / V15 Type Registry Parser"]
    B --> E["Runtime Wasm Entrypoint Decompiler"]
    B --> F["Runtime Upgrade Diff Engine"]
    
    C --> C1["Compact Integer & Variant Decoding"]
    C --> C2["PortableRegistry Type Index (Types 0..N)"]
    
    D --> D1["Pallet Catalog (Pallet Index, Name, Call Enum)"]
    D --> D2["Storage Entry Key Derivation (Twox128 / Blake2)"]
    D --> D3["Extrinsic Call Decoder (PalletIdx, CallIdx, Args)"]
    
    E --> E1[":code WebAssembly Module Analysis (#2298)"]
    E --> E2["Runtime API Entrypoints (Core, Metadata, BlockBuilder)"]
    
    F --> F1["Metadata Version Comparison (Pre vs Post Upgrade)"]
    F --> F2["Storage Migration & Call Signature Breaking Changes"]
    
    C & D & E & F --> G["Synthesized Pallet Specification, Extrinsic Tree & Upgrade Diff Report"]

Technical Capabilities & Modules

SCALE Codec & PortableRegistry Parser:
- Decode SCALE primitive types (Compact integers, booleans, fixed arrays, vectors, options, results).
- Parse Metadata V14 and V15 PortableRegistry: reconstruct user-defined structs, enums, tuples, and bit sequences.
Pallet Catalog & Storage Key Derivation:
- Enumerate all pallets registered in the runtime: name, index, calls, storage, events, errors, constants.
- Calculate Substrate storage keys: apply hasher algorithms (Twox128(PalletName) ++ Twox128(StorageName) ++ Hasher(Key) using Blake2_128Concat, Twox64Concat, Identity).
Extrinsic (Transaction) Dissection:
- Parse signed and unsigned extrinsic payloads: version header, signature, nonce, tip, era (mortality), and call data.
- Resolve pallet index and call index into human-readable function names and typed parameter values.
Runtime Wasm Introspection:
- Ingest runtime WebAssembly blobs (:code) utilizing REA's WASM engine (#2298, #2350).
- Trace exported Substrate runtime API functions: Core_version, Metadata_metadata, TaggedTransactionQueue_validate_transaction.
Semantic Runtime Upgrade Diffing:
- Compare two metadata snapshots across runtime upgrades.
- Identify added/removed/renamed pallets.
- Detect breaking changes in storage item types and extrinsic call parameter schemas.

Acceptance Criteria

Automated parsing and decoding of Substrate SCALE Metadata V14 and V15 payloads.
Complete reconstruction of pallet catalogs, call functions, and storage item keys.
Accurate decoding of raw extrinsics into human-readable JSON call trees.
Introspection of runtime Wasm modules and export of runtime API signatures.
Semantic diff reporting structural changes between consecutive runtime versions.

Complements #2350 (Inspect WebAssembly binaries)
Complements #2298 (Decompile WebAssembly modules)
Complements #3621 (Unknown binary file-format inference)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3738 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Blockchain RE](https://img.shields.io/badge/domain-blockchain_RE-orange.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context Polkadot and Substrate-based blockchains employ a unique runtime architecture: the entire blockchain state transition logic is compiled into a WebAssembly binary (`runtime.wasm`) stored directly within the on-chain storage. When a blockchain upgrade occurs, the runtime Wasm is overwritten on-chain (forkless runtime upgrade), fundamentally altering pallets, storage structures, extrinsics, and dispatch calls. Reverse engineering a Substrate chain, node snapshot, or transaction requires decoding: 1. **SCALE Codec (Simple Concatenated Aggregate Little-Endian)**: Compact binary serialization format used across Substrate for types, storage entries, and network packets. 2. **SCALE Runtime Metadata (V14 & V15)**: Self-describing type registry (`PortableRegistry`) and pallet manifests embedded in the runtime. Defines every pallet, call function (`Call`), storage entry, event, custom error, and constant. 3. **Runtime Wasm Blob (`:code`)**: Compressed or uncompressed WebAssembly module implementing runtime API entrypoints (`Core_version`, `Metadata_metadata`, `BlockBuilder_apply_extrinsic`). 4. **Pallet Architecture & Extrinsics**: Modularity decomposed into individual pallets (`System`, `Balances`, `Staking`, `Governance`). Extrinsics (transactions) encoded as pallet index + call index + arguments. 5. **Runtime Upgrade Diffing**: Comparing Metadata V14/V15 and Wasm binaries between Block $N$ and Block $N+1$ to determine what pallets were added, storage layouts migrated, or call signatures modified. REA needs a dedicated Substrate engine capable of parsing SCALE metadata, disassembling runtime Wasm entrypoints, decoding extrinsics, and performing semantic diffs across runtime upgrades. ### Proposed Architecture ```mermaid flowchart TD A["Substrate Artifact (Metadata Hex / runtime.wasm / Block Extrinsic)"] --> B["Substrate / Polkadot Semantic Engine"] B --> C["SCALE Codec Deserializer"] B --> D["Metadata V14 / V15 Type Registry Parser"] B --> E["Runtime Wasm Entrypoint Decompiler"] B --> F["Runtime Upgrade Diff Engine"] C --> C1["Compact Integer & Variant Decoding"] C --> C2["PortableRegistry Type Index (Types 0..N)"] D --> D1["Pallet Catalog (Pallet Index, Name, Call Enum)"] D --> D2["Storage Entry Key Derivation (Twox128 / Blake2)"] D --> D3["Extrinsic Call Decoder (PalletIdx, CallIdx, Args)"] E --> E1[":code WebAssembly Module Analysis (#2298)"] E --> E2["Runtime API Entrypoints (Core, Metadata, BlockBuilder)"] F --> F1["Metadata Version Comparison (Pre vs Post Upgrade)"] F --> F2["Storage Migration & Call Signature Breaking Changes"] C & D & E & F --> G["Synthesized Pallet Specification, Extrinsic Tree & Upgrade Diff Report"] ``` ### Technical Capabilities & Modules 1. **SCALE Codec & PortableRegistry Parser:** - Decode SCALE primitive types (Compact integers, booleans, fixed arrays, vectors, options, results). - Parse Metadata V14 and V15 `PortableRegistry`: reconstruct user-defined structs, enums, tuples, and bit sequences. 2. **Pallet Catalog & Storage Key Derivation:** - Enumerate all pallets registered in the runtime: name, index, calls, storage, events, errors, constants. - Calculate Substrate storage keys: apply hasher algorithms (`Twox128(PalletName) ++ Twox128(StorageName) ++ Hasher(Key)` using Blake2_128Concat, Twox64Concat, Identity). 3. **Extrinsic (Transaction) Dissection:** - Parse signed and unsigned extrinsic payloads: version header, signature, nonce, tip, era (mortality), and call data. - Resolve pallet index and call index into human-readable function names and typed parameter values. 4. **Runtime Wasm Introspection:** - Ingest runtime WebAssembly blobs (`:code`) utilizing REA's WASM engine (#2298, #2350). - Trace exported Substrate runtime API functions: `Core_version`, `Metadata_metadata`, `TaggedTransactionQueue_validate_transaction`. 5. **Semantic Runtime Upgrade Diffing:** - Compare two metadata snapshots across runtime upgrades. - Identify added/removed/renamed pallets. - Detect breaking changes in storage item types and extrinsic call parameter schemas. ### Acceptance Criteria - [ ] Automated parsing and decoding of Substrate SCALE Metadata V14 and V15 payloads. - [ ] Complete reconstruction of pallet catalogs, call functions, and storage item keys. - [ ] Accurate decoding of raw extrinsics into human-readable JSON call trees. - [ ] Introspection of runtime Wasm modules and export of runtime API signatures. - [ ] Semantic diff reporting structural changes between consecutive runtime versions. ### Related Issues & Dependencies - Complements #2350 (Inspect WebAssembly binaries) - Complements #2298 (Decompile WebAssembly modules) - Complements #3621 (Unknown binary file-format inference)

bsakaguchi referenced this issue

2026-10-17 21:11:29 +00:00

[PR #1447] [MERGED] feat(harmony): project HarmonyOS package inventory evidence #2893

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1447

No description provided.

Rows
Columns