[GH-ISSUE #3708] [Feature] 2D tile-grid MMO asset reverse engineering: OTB/OTBM binary maps, versioned sprite catalogs, client UI modules and packet framing schemas #1419

New issue

Open

opened 2026-10-16 21:05:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:17 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3708

Problem Statement & Context

Tile-grid based 2D multiplayer virtual worlds operate using specialized node-based binary map formats, versioned item catalogs, modular client interfaces, and state-synchronizing network protocols:

Node-Based Map Serialization (.otbm / .otb):
- Binary tree node container format governed by special escape markers: NODE_START (0xFE), NODE_END (0xFF), and escape byte 0xFD.
- .otb (Item Database): Stores client-server item metadata (flags: unpassable, block path, pickupable, stackable, light source, writeable).
- .otbm (Binary Map): Hierarchical node tree containing map header, map attributes, description, tile areas (OTBM_TILE_AREA), tiles, and nested item containers.
Sprite & Data Catalogs (.spr, .dat):
- .dat: Item, creature, effect, and projectile attribute definitions (width, height, crop sizes, blend frames, animation frames).
- .spr: 32x32 pixel indexed or RGBA sprite chunks preceded by signature checksums.
Modular Client Architecture (OTClient): Open, script-driven client environments configured with .otml (markup language), .otui (user interface definitions), and modular Lua controllers.
Network Protocol Framing & State Synchronization: Custom framing, packet length headers, XTEA / RSA payload encryption wrappers, opcode byte dispatch tables, and position-based tile map synchronization.

REA lacks native tooling to parse OTBM node-based binary maps, decode .dat/.spr versioned catalogs, and model recorded client network frames.

Proposed Architecture

flowchart TD
    A["Map (.otbm) / Catalog (.spr, .dat) / Client Module / Network Capture"] --> B["2D Grid MMO Semantic Engine"]
    B --> C["Node-Based Stream Deserializer (.otbm / .otb)"]
    B --> D["Sprite Catalog & Item Attribute Parser"]
    B --> E["Client UI & Protocol Analyzer"]
    
    C --> C1["NODE_START / NODE_END Tree Traversal"]
    C --> C2["Escape Byte Unpacking (0xFD)"]
    C --> C3["Tile Area & Coordinate Slicing (X, Y, Z Z-axis levels)"]
    
    D --> D1[".dat Item Type & Animation Frame Definitions"]
    D --> D2[".spr 32x32 Pixel Surface Extraction (PNG)"]
    
    E --> E1[".otml & .otui Layout Tree Deserialization"]
    E --> E2["Client-Server Protocol Framing & Opcode Mapping"]
    
    C & D & E --> F["Synthesized Tile Map Layout, Asset Catalog & Protocol Blueprint"]

Technical Capabilities & Modules

Escape-Encoded Node Stream Deserialization:
- Parse escape-encoded binary node trees (0xFE start, 0xFF end, 0xFD escape).
- Parse .otb item definition tables: flags (movement cost, fluid container, light parameters).
- Parse .otbm binary maps: root headers, area coordinates, tile nodes, item instances, and creature spawn metadata.
Sprite & Item Catalog Extraction (.spr, .dat):
- Validate version signatures and sprite counts in .spr catalogs.
- Extract 32x32 RGBA / indexed sprite surfaces and stitch multi-tile objects (e.g. 64x64, 96x96 structures).
- Reconstruct animated keyframes from .dat frame timing tables.
Modular Client UI & Layout Parsing:
- Parse .otml (Object Tree Markup Language) and .otui UI definitions.
- Correlate UI button triggers and window definitions with client Lua event callbacks.
Network Stream Framing & Opcode Correlation:
- Dissect offline recorded session captures (.pcap or packet dumps).
- Identify packet length headers and strip transport framing.
- Map opcode bytes to known client handlers (e.g., login, map description, tile update, creature walk, player chat) using declarative protocol schemas.

Acceptance Criteria

Robust parsing of .otbm binary map trees without memory exhaustion on large multi-floor maps.
Safe deserialization of .otb item metadata tables.
Extraction of 32x32 sprite sheets from .spr and correlation with .dat item attributes.
Parsing of .otml and .otui client user interface layouts.
Declarative decoding of recorded packet streams matching opcodes to client handler functions.

Complements #3621 (Unknown binary file-format inference)
Complements #3699 (Lua and LuaJIT bytecode reverse engineering)
Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3708 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context Tile-grid based 2D multiplayer virtual worlds operate using specialized node-based binary map formats, versioned item catalogs, modular client interfaces, and state-synchronizing network protocols: 1. **Node-Based Map Serialization (`.otbm` / `.otb`)**: - Binary tree node container format governed by special escape markers: `NODE_START` (`0xFE`), `NODE_END` (`0xFF`), and escape byte `0xFD`. - `.otb` (Item Database): Stores client-server item metadata (flags: unpassable, block path, pickupable, stackable, light source, writeable). - `.otbm` (Binary Map): Hierarchical node tree containing map header, map attributes, description, tile areas (`OTBM_TILE_AREA`), tiles, and nested item containers. 2. **Sprite & Data Catalogs (`.spr`, `.dat`)**: - `.dat`: Item, creature, effect, and projectile attribute definitions (width, height, crop sizes, blend frames, animation frames). - `.spr`: 32x32 pixel indexed or RGBA sprite chunks preceded by signature checksums. 3. **Modular Client Architecture (OTClient)**: Open, script-driven client environments configured with `.otml` (markup language), `.otui` (user interface definitions), and modular Lua controllers. 4. **Network Protocol Framing & State Synchronization**: Custom framing, packet length headers, XTEA / RSA payload encryption wrappers, opcode byte dispatch tables, and position-based tile map synchronization. REA lacks native tooling to parse OTBM node-based binary maps, decode `.dat`/`.spr` versioned catalogs, and model recorded client network frames. ### Proposed Architecture ```mermaid flowchart TD A["Map (.otbm) / Catalog (.spr, .dat) / Client Module / Network Capture"] --> B["2D Grid MMO Semantic Engine"] B --> C["Node-Based Stream Deserializer (.otbm / .otb)"] B --> D["Sprite Catalog & Item Attribute Parser"] B --> E["Client UI & Protocol Analyzer"] C --> C1["NODE_START / NODE_END Tree Traversal"] C --> C2["Escape Byte Unpacking (0xFD)"] C --> C3["Tile Area & Coordinate Slicing (X, Y, Z Z-axis levels)"] D --> D1[".dat Item Type & Animation Frame Definitions"] D --> D2[".spr 32x32 Pixel Surface Extraction (PNG)"] E --> E1[".otml & .otui Layout Tree Deserialization"] E --> E2["Client-Server Protocol Framing & Opcode Mapping"] C & D & E --> F["Synthesized Tile Map Layout, Asset Catalog & Protocol Blueprint"] ``` ### Technical Capabilities & Modules 1. **Escape-Encoded Node Stream Deserialization:** - Parse escape-encoded binary node trees (`0xFE` start, `0xFF` end, `0xFD` escape). - Parse `.otb` item definition tables: flags (movement cost, fluid container, light parameters). - Parse `.otbm` binary maps: root headers, area coordinates, tile nodes, item instances, and creature spawn metadata. 2. **Sprite & Item Catalog Extraction (`.spr`, `.dat`):** - Validate version signatures and sprite counts in `.spr` catalogs. - Extract 32x32 RGBA / indexed sprite surfaces and stitch multi-tile objects (e.g. 64x64, 96x96 structures). - Reconstruct animated keyframes from `.dat` frame timing tables. 3. **Modular Client UI & Layout Parsing:** - Parse `.otml` (Object Tree Markup Language) and `.otui` UI definitions. - Correlate UI button triggers and window definitions with client Lua event callbacks. 4. **Network Stream Framing & Opcode Correlation:** - Dissect offline recorded session captures (`.pcap` or packet dumps). - Identify packet length headers and strip transport framing. - Map opcode bytes to known client handlers (e.g., login, map description, tile update, creature walk, player chat) using declarative protocol schemas. ### Acceptance Criteria - [ ] Robust parsing of `.otbm` binary map trees without memory exhaustion on large multi-floor maps. - [ ] Safe deserialization of `.otb` item metadata tables. - [ ] Extraction of 32x32 sprite sheets from `.spr` and correlation with `.dat` item attributes. - [ ] Parsing of `.otml` and `.otui` client user interface layouts. - [ ] Declarative decoding of recorded packet streams matching opcodes to client handler functions. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3699 (Lua and LuaJIT bytecode reverse engineering) - Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

bsakaguchi referenced this issue

2026-10-17 21:11:15 +00:00

[PR #1419] [MERGED] fix(ghidra): clear wrong no-return flags before serving a session #2871

bsakaguchi referenced this issue

2026-10-31 00:56:06 +00:00

[PR #1863] [MERGED] fix(windows): bring the real Windows Ghidra verifiers up to date #4859

bsakaguchi referenced this issue

2026-10-31 01:14:36 +00:00

[PR #4892] fix(ghidra): treat demangled C++ runtime imports as returning #6460