SB super-bundles, EBX typed reflection trees and RES resource graphs #1420

New issue

Open

opened 2026-10-16 21:05:17 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:17 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3710

Problem Statement & Context

High-performance distributed rendering and simulation engines utilize a unified content delivery and asset runtime architecture:

Catalog & Chunk Storage (CAT / CAS): Decoupled content distribution where catalogs (.cat) map 128-bit or 160-bit SHA-1 content hashes to physical data chunk slices stored across content-addressable storage files (cas_01.cas, cas_02.cas, etc.).
Table of Contents & Super-Bundles (.toc / .sb): Multi-level hierarchical bundles declaring bundled chunks, streaming chunks, and shared resource dictionaries. Supports LZ4, Zlib, and Oodle chunk-level compression.
Engine Binary XML (EBX): Type-safe, RTTI-driven binary reflection serialization format storing all game logic, level layout, entity schemas, UI screens, and physics definitions. Contains internal class descriptors, field offsets, guid references, and strongly-typed property arrays.
Compiled Resource Headers (RES): Low-level platform hardware resources (ShaderBlockDepot, MeshSet, Texture, AntAnimation) linked to EBX instances via resource GUIDs.

REA lacks native capabilities to resolve CAT/CAS chunk references, unpack super-bundles, and deserialize typed EBX reflection trees.

Proposed Architecture

flowchart TD
    A["Engine Directory (CAT / CAS / .toc / .sb bundles)"] --> B["CAT/CAS & EBX Semantic Engine"]
    B --> C["Catalog & Super-Bundle Resolver"]
    B --> D["Chunk Slicer & Decompressor (CAS)"]
    B --> E["EBX Object Reflection Deserializer"]
    
    C --> C1[".cat Hash Tables (160-bit SHA-1 Chunk IDs)"]
    C --> C2[".toc / .sb Super-Bundle Hierarchy Parsing"]
    
    D --> D1["cas_NN.cas Byte Offset Slicing"]
    D --> D2["Chunk Decompression (Zlib / LZ4 / Oodle)"]
    
    E --> E1["EBX Header, Class Descriptors & Field Layouts"]
    E --> E2["GUID Reference Resolution & Instance Hierarchy"]
    E --> E3["RES Resource Metadata Linking"]
    
    C & D & E --> F["Synthesized Asset Catalog, EBX Logic Graph & Entity Schema Tree"]

Technical Capabilities & Modules

CAT / CAS Content-Addressable Storage Parsing:
- Parse catalog files (.cat): extract chunk SHA-1 digests, target cas_NN.cas file indices, byte offsets, and chunk byte sizes.
- Slice arbitrary data chunks from physical CAS containers without needing full-archive extractions.
Super-Bundle & TOC Deserialization:
- Parse .toc and .sb container headers, distinguishing between base bundles, chunk bundles, and delta patch bundles.
- Decompress multi-block compressed streams using Zlib, LZ4, or Oodle.
- Extract raw EBX payloads, RES payloads, and streaming chunk references.
EBX Reflection Deserialization:
- Parse EBX binary structure: type metadata dictionary, field descriptor tables, and instance payload blocks.
- Reconstruct strongly-typed object trees (classes, structs, arrays, primitives, enum flags).
- Resolve internal object GUIDs and cross-bundle external asset pointers.
Resource Linkage & Asset Topology:
- Map EBX entities to corresponding RES hardware assets (e.g. Texture -> streaming chunk IDs, MeshSet -> LOD geometry chunks).
- Export extracted EBX trees to clean, human-readable JSON formats.

Acceptance Criteria

Parsing of .cat catalogs and successful chunk lookup across multi-file cas_*.cas volumes.
Safe unpacking and decompression of .toc / .sb super-bundles.
Full deserialization of EBX binary files into typed JSON object models.
Resolution of inter-asset GUID references across separate bundles.
Generation of an end-to-end asset dependency map.

Complements #3621 (Unknown binary file-format inference)
Complements #3624 (Native C++ semantic reconstruction: RTTI, vtables)
Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3710 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context High-performance distributed rendering and simulation engines utilize a unified content delivery and asset runtime architecture: 1. **Catalog & Chunk Storage (`CAT` / `CAS`)**: Decoupled content distribution where catalogs (`.cat`) map 128-bit or 160-bit SHA-1 content hashes to physical data chunk slices stored across content-addressable storage files (`cas_01.cas`, `cas_02.cas`, etc.). 2. **Table of Contents & Super-Bundles (`.toc` / `.sb`)**: Multi-level hierarchical bundles declaring bundled chunks, streaming chunks, and shared resource dictionaries. Supports LZ4, Zlib, and Oodle chunk-level compression. 3. **Engine Binary XML (EBX)**: Type-safe, RTTI-driven binary reflection serialization format storing all game logic, level layout, entity schemas, UI screens, and physics definitions. Contains internal class descriptors, field offsets, guid references, and strongly-typed property arrays. 4. **Compiled Resource Headers (`RES`)**: Low-level platform hardware resources (`ShaderBlockDepot`, `MeshSet`, `Texture`, `AntAnimation`) linked to EBX instances via resource GUIDs. REA lacks native capabilities to resolve CAT/CAS chunk references, unpack super-bundles, and deserialize typed EBX reflection trees. ### Proposed Architecture ```mermaid flowchart TD A["Engine Directory (CAT / CAS / .toc / .sb bundles)"] --> B["CAT/CAS & EBX Semantic Engine"] B --> C["Catalog & Super-Bundle Resolver"] B --> D["Chunk Slicer & Decompressor (CAS)"] B --> E["EBX Object Reflection Deserializer"] C --> C1[".cat Hash Tables (160-bit SHA-1 Chunk IDs)"] C --> C2[".toc / .sb Super-Bundle Hierarchy Parsing"] D --> D1["cas_NN.cas Byte Offset Slicing"] D --> D2["Chunk Decompression (Zlib / LZ4 / Oodle)"] E --> E1["EBX Header, Class Descriptors & Field Layouts"] E --> E2["GUID Reference Resolution & Instance Hierarchy"] E --> E3["RES Resource Metadata Linking"] C & D & E --> F["Synthesized Asset Catalog, EBX Logic Graph & Entity Schema Tree"] ``` ### Technical Capabilities & Modules 1. **CAT / CAS Content-Addressable Storage Parsing:** - Parse catalog files (`.cat`): extract chunk SHA-1 digests, target `cas_NN.cas` file indices, byte offsets, and chunk byte sizes. - Slice arbitrary data chunks from physical CAS containers without needing full-archive extractions. 2. **Super-Bundle & TOC Deserialization:** - Parse `.toc` and `.sb` container headers, distinguishing between base bundles, chunk bundles, and delta patch bundles. - Decompress multi-block compressed streams using Zlib, LZ4, or Oodle. - Extract raw EBX payloads, RES payloads, and streaming chunk references. 3. **EBX Reflection Deserialization:** - Parse EBX binary structure: type metadata dictionary, field descriptor tables, and instance payload blocks. - Reconstruct strongly-typed object trees (classes, structs, arrays, primitives, enum flags). - Resolve internal object GUIDs and cross-bundle external asset pointers. 4. **Resource Linkage & Asset Topology:** - Map EBX entities to corresponding `RES` hardware assets (e.g. `Texture` -> streaming chunk IDs, `MeshSet` -> LOD geometry chunks). - Export extracted EBX trees to clean, human-readable JSON formats. ### Acceptance Criteria - [ ] Parsing of `.cat` catalogs and successful chunk lookup across multi-file `cas_*.cas` volumes. - [ ] Safe unpacking and decompression of `.toc` / `.sb` super-bundles. - [ ] Full deserialization of EBX binary files into typed JSON object models. - [ ] Resolution of inter-asset GUID references across separate bundles. - [ ] Generation of an end-to-end asset dependency map. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3624 (Native C++ semantic reconstruction: RTTI, vtables) - Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

bsakaguchi referenced this issue

2026-10-17 21:11:16 +00:00

[PR #1420] [MERGED] docs: correct stale logging, Ghidra host, and example path statements #2873

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1420

No description provided.

Rows
Columns