ADT world maps, M2 models, BLP textures and TOC addon graphs #1416

New issue

Open

opened 2026-10-16 21:05:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:15 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3704

Problem Statement & Context

High-scale online fantasy virtual worlds and client-server architectures have transitioned across two major generations of proprietary virtual storage systems:

MoPaQ Containers (.mpq): Multi-archive priority chains, block/hash tables (using Jenkins one-at-a-time / custom encryption hash algorithms), sector-based compression (Zlib, Bzip2, LZMA, Huffman, PKWARE Implode), and locale masks.
Content Addressable Storage Containers (CASC): Modern distributed storage architecture utilizing content-addressable storage keys (EKey encoding keys and CKey content keys), local index files (.idx), data archives (data.000–data.NNN), encoding files, root files, and downloaded install manifests.
Structured Data Tables (DBC / DB2 / WDB): Binary schema tables containing client-side item definitions, spell tables, map coordinates, and localized string blocks (supporting various format revisions: WDBC, WDB2, WDB5, WDB6, WDC1, WDC2, WDC3, WDC4 with bit-packed sparse columns).
Terrain & Spatial Geometry (WDT, ADT, WMO, M2): World definition tables, chunked landscape terrain grids with heightmaps and alpha splatting, world model objects, and skeletal animated meshes.
Texture Formats & Addon UI Frameworks (.blp, .toc): Compressed image textures (DXT compression, paletted color maps, mipmaps) and user interface modification packages governed by .toc manifests, XML UI layouts, and Lua script controllers.

REA lacks native support for reading these hybrid virtual filesystems and interpreting their internal binary data structures.

Proposed Architecture

flowchart TD
    A["Game Directory (MPQ Archives / CASC data directory)"] --> B["Virtual File System & Asset Engine"]
    B --> C["Storage Driver (MPQ Unpacker / CASC Index Resolver)"]
    B --> D["Data Table Deserializer (DBC / DB2)"]
    B --> E["Asset Pipeline & Addon Inspector"]
    
    C --> C1["MPQ Header, Hash/Block Tables & Sector Decryption"]
    C --> C2["CASC Root, Encoding & .idx Key Resolution"]
    
    D --> D1["WDBC / WDB2 / WDCx Header & Field Layout Parser"]
    D --> D2["Bit-Packed & Paletted Column Deserialization"]
    D --> D3["Localized String Block Indexing"]
    
    E --> E1[".blp Texture Extractor (DXT / Paletted -> PNG)"]
    E --> E2[".toc Addon Manifests & Lua/XML UI Dependency Graph"]
    
    C & D & E --> F["Synthesized Storage Manifest, Data Schemas & Addon Topology"]

Technical Capabilities & Modules

MoPaQ Archive Deserialization (.mpq):
- Parse MPQ headers (versions 1 through 4), extended block tables, and HET/BET tables.
- Implement hash table lookup with custom polynomial encryption keys.
- Decompress multi-algorithm sectors: Zlib, Bzip2, LZMA, and PKWARE DCL.
CASC Storage Ingestion:
- Parse local CASC storage index files (data/*.idx).
- Decode root and encoding files mapping file paths / FileDataIDs to Content Keys (CKey) and Encoding Keys (EKey).
- Reconstruct virtual file paths from community listfiles or embedded path hashes without executing client binaries.
Client Database Table Parser (DBC / DB2):
- Parse legacy WDBC and modern WDC1–WDC4 file formats.
- Decode column types: immediate integer, float, bit-packed paletted arrays, string offsets, and relation records.
- Export structured tables to clean JSON or CSV schemas.
Image & Texture Decoding (.blp):
- Support BLP1 and BLP2 image types.
- Decode DXT1, DXT3, DXT5, and paletted color formats into standard RGBA images.
Client Addon & UI Architecture Analysis:
- Parse .toc (Table of Contents) addon descriptor files: interface versions, titles, saved variables, dependencies (## Dependencies, ## OptionalDeps).
- Reconstruct XML user interface element hierarchies and their corresponding Lua script handlers.

Acceptance Criteria

Static extraction of files from multi-part .mpq archives with sector decompression.
Safe resolution of content files from local CASC data directories using .idx index tables.
Schema parsing and tabular record extraction for WDBC and modern WDCx binary tables.
Conversion of .blp textures to standard PNG/DDS images.
Parsing of .toc addon manifests and construction of an addon dependency graph.

Complements #3621 (Unknown binary file-format inference)
Complements #3699 (Lua and LuaJIT bytecode reverse engineering)
Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3704 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context High-scale online fantasy virtual worlds and client-server architectures have transitioned across two major generations of proprietary virtual storage systems: 1. **MoPaQ Containers (`.mpq`)**: Multi-archive priority chains, block/hash tables (using Jenkins one-at-a-time / custom encryption hash algorithms), sector-based compression (Zlib, Bzip2, LZMA, Huffman, PKWARE Implode), and locale masks. 2. **Content Addressable Storage Containers (CASC)**: Modern distributed storage architecture utilizing content-addressable storage keys (`EKey` encoding keys and `CKey` content keys), local index files (`.idx`), data archives (`data.000`–`data.NNN`), encoding files, root files, and downloaded install manifests. 3. **Structured Data Tables (`DBC` / `DB2` / `WDB`)**: Binary schema tables containing client-side item definitions, spell tables, map coordinates, and localized string blocks (supporting various format revisions: `WDBC`, `WDB2`, `WDB5`, `WDB6`, `WDC1`, `WDC2`, `WDC3`, `WDC4` with bit-packed sparse columns). 4. **Terrain & Spatial Geometry (`WDT`, `ADT`, `WMO`, `M2`)**: World definition tables, chunked landscape terrain grids with heightmaps and alpha splatting, world model objects, and skeletal animated meshes. 5. **Texture Formats & Addon UI Frameworks (`.blp`, `.toc`)**: Compressed image textures (DXT compression, paletted color maps, mipmaps) and user interface modification packages governed by `.toc` manifests, XML UI layouts, and Lua script controllers. REA lacks native support for reading these hybrid virtual filesystems and interpreting their internal binary data structures. ### Proposed Architecture ```mermaid flowchart TD A["Game Directory (MPQ Archives / CASC data directory)"] --> B["Virtual File System & Asset Engine"] B --> C["Storage Driver (MPQ Unpacker / CASC Index Resolver)"] B --> D["Data Table Deserializer (DBC / DB2)"] B --> E["Asset Pipeline & Addon Inspector"] C --> C1["MPQ Header, Hash/Block Tables & Sector Decryption"] C --> C2["CASC Root, Encoding & .idx Key Resolution"] D --> D1["WDBC / WDB2 / WDCx Header & Field Layout Parser"] D --> D2["Bit-Packed & Paletted Column Deserialization"] D --> D3["Localized String Block Indexing"] E --> E1[".blp Texture Extractor (DXT / Paletted -> PNG)"] E --> E2[".toc Addon Manifests & Lua/XML UI Dependency Graph"] C & D & E --> F["Synthesized Storage Manifest, Data Schemas & Addon Topology"] ``` ### Technical Capabilities & Modules 1. **MoPaQ Archive Deserialization (`.mpq`):** - Parse MPQ headers (versions 1 through 4), extended block tables, and HET/BET tables. - Implement hash table lookup with custom polynomial encryption keys. - Decompress multi-algorithm sectors: Zlib, Bzip2, LZMA, and PKWARE DCL. 2. **CASC Storage Ingestion:** - Parse local CASC storage index files (`data/*.idx`). - Decode root and encoding files mapping file paths / FileDataIDs to Content Keys (`CKey`) and Encoding Keys (`EKey`). - Reconstruct virtual file paths from community listfiles or embedded path hashes without executing client binaries. 3. **Client Database Table Parser (`DBC` / `DB2`):** - Parse legacy `WDBC` and modern `WDC1`–`WDC4` file formats. - Decode column types: immediate integer, float, bit-packed paletted arrays, string offsets, and relation records. - Export structured tables to clean JSON or CSV schemas. 4. **Image & Texture Decoding (`.blp`):** - Support BLP1 and BLP2 image types. - Decode DXT1, DXT3, DXT5, and paletted color formats into standard RGBA images. 5. **Client Addon & UI Architecture Analysis:** - Parse `.toc` (Table of Contents) addon descriptor files: interface versions, titles, saved variables, dependencies (`## Dependencies`, `## OptionalDeps`). - Reconstruct XML user interface element hierarchies and their corresponding Lua script handlers. ### Acceptance Criteria - [ ] Static extraction of files from multi-part `.mpq` archives with sector decompression. - [ ] Safe resolution of content files from local CASC data directories using `.idx` index tables. - [ ] Schema parsing and tabular record extraction for `WDBC` and modern `WDCx` binary tables. - [ ] Conversion of `.blp` textures to standard PNG/DDS images. - [ ] Parsing of `.toc` addon manifests and construction of an addon dependency graph. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3699 (Lua and LuaJIT bytecode reverse engineering) - Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

bsakaguchi referenced this issue

2026-10-16 21:05:48 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

bsakaguchi referenced this issue

2026-10-16 21:05:49 +00:00

[GH-ISSUE #3796] [Tracker] Open feature requests by area (October 2026) #1471

bsakaguchi referenced this issue

2026-10-17 21:01:20 +00:00

[GH-ISSUE #1416] [Feature] Support the official Hex-Rays IDA MCP server as an analysis provider #485

bsakaguchi referenced this issue

2026-10-31 01:14:20 +00:00

[PR #4813] feat(ida): support the official Hex-Rays IDA MCP headlessly #6421