BA2 containers, FormID references and Papyrus PEX bytecode #1417

New issue

Open

opened 2026-10-16 21:05:16 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:16 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3706

Problem Statement & Context

Extensible open-world RPG engines rely on a modular, multi-layer plugin and master file architecture spanning multiple generations of desktop and console titles:

Master & Plugin Records (.esm, .esp, .esl): Hierarchical binary databases structured as 4-character Group records (GRUP) containing specific Form records (e.g. TES4, GMST, GLOB, KYWD, SPEL, WEAP, NPC_, CELL, WRLD, DIAL, QUST). Each record is identified by a unique 32-bit FormID.
FormID Cross-Referencing & Load Order Resolution: Records reference other entities via FormIDs whose upper 8 bits (or 12 bits in ESL lightweight plugins) resolve dynamically based on the active plugin load order index (plugins.txt).
Asset Archive Containers (.bsa / .ba2):
- Legacy BSA: Folder record tables, file record tables, name hashes, and Zlib-compressed data blocks.
- Modern BA2: File type signatures (General vs GNRL vs DX10 texture streams), string table headers, and compression headers.
Compiled Papyrus Script Bytecode (.pex): Custom object-oriented virtual machine executing compiled .pex scripts attached to quests, items, scenes, and magic effects. Scripts define classes, properties, variables, state machines, and event handlers.
NIF Geometry & Animation Linking: 3D scene graphs (.nif) referencing skeleton bones, shader textures, collision hulls, and animation controllers.

REA requires a dedicated engine to parse plugin record hierarchies, resolve FormID cross-references across multi-file load orders, disassemble Papyrus .pex bytecode, and inspect .bsa/.ba2 archives.

Proposed Architecture

flowchart TD
    A["Plugin File (.esm/.esp/.esl) / Archive (.bsa/.ba2) / Script (.pex)"] --> B["Plugin & Modular Asset Engine"]
    B --> C["Archive Container Unpacker (BSA / BA2)"]
    B --> D["Record & FormID Hierarchy Parser"]
    B --> E["Papyrus Bytecode Decompiler (.pex)"]
    
    C --> C1["BSA Directory & Hash Tables (Zlib Slices)"]
    C --> C2["BA2 General & DX10 Chunk Deserializer"]
    
    D --> D1["GRUP & Record Header Parsing (4-byte Signatures)"]
    D --> D2["Subrecord Decoding (EDID, FULL, DATA, OBND)"]
    D --> D3["FormID Reference Graph & Load Order Resolution"]
    
    E --> E1[".pex Header, String Table & User Flags"]
    E --> E2["Class Definitions, Properties & State Machines"]
    E --> E3["Opcode Stream Disassembly & Event Function Control Flow"]
    
    C & D & E --> F["Synthesized Plugin Dependency Tree, Reconstructed Scripts & Asset Manifest"]

Technical Capabilities & Modules

BSA and BA2 Archive Extraction:
- Parse legacy BSA headers (magic 0x00041534), folder/file tables, filename hashes, and compression flags.
- Parse modern BA2 headers (magic BTDX), separate General archives from DX10 texture stream archives, and extract compressed asset files.
Hierarchical Record Parsing (.esm, .esp, .esl):
- Traverse GRUP containers, handling group types: top-level record type, in-world children, interior sub-cells, and topic children.
- Parse record headers: FormID, flags (compressed, master, deleted), data version, and revision numbers.
- Decompress Zlib-compressed records and parse subrecords (4-character tags + 16-bit/32-bit length fields).
FormID Cross-Reference & Conflict Resolution:
- Index all defined FormIDs and build a reference table mapping which records reference or override others.
- Resolve relative FormID indexing against master dependencies declared in the TES4 header (MAST subrecords).
- Detect record override chains (e.g. Weapon balance tweaks or dialogue overrides).
Papyrus Bytecode Disassembly & Decompilation (.pex):
- Parse .pex binary structure: magic (0xFA57C0DE), compiler version, compilation timestamp, source file name, user name, machine name.
- Decode string tables and type tables.
- Reconstruct class definitions, parent inheritance, declared properties, and variable types.
- Disassemble Papyrus opcodes (callmethod, callparent, assign, return, cast, jumpt, jumpf) and decompile state machine transitions into readable Papyrus source code.
Dialogue & Quest Flow Reconstruction:
- Parse QUST (Quest) and DIAL / INFO (Dialogue) records.
- Reconstruct quest stage state machines, objective log entries, and dialogue choice branches.

Acceptance Criteria

Parsing and safe extraction of assets from both .bsa and .ba2 archives.
Complete traversal of GRUP, record, and subrecord hierarchies across .esm, .esp, and .esl plugins.
Dynamic FormID mapping linking records to their parent master files.
Disassembly and source code decompilation of compiled .pex Papyrus script bytecodes.
Generation of an interactive quest and dialogue flowchart.

Complements #3621 (Unknown binary file-format inference)
Complements #3624 (Native C++ semantic reconstruction: RTTI, vtables)
Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3706 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context Extensible open-world RPG engines rely on a modular, multi-layer plugin and master file architecture spanning multiple generations of desktop and console titles: 1. **Master & Plugin Records (`.esm`, `.esp`, `.esl`)**: Hierarchical binary databases structured as 4-character Group records (`GRUP`) containing specific Form records (e.g. `TES4`, `GMST`, `GLOB`, `KYWD`, `SPEL`, `WEAP`, `NPC_`, `CELL`, `WRLD`, `DIAL`, `QUST`). Each record is identified by a unique 32-bit FormID. 2. **FormID Cross-Referencing & Load Order Resolution**: Records reference other entities via FormIDs whose upper 8 bits (or 12 bits in ESL lightweight plugins) resolve dynamically based on the active plugin load order index (`plugins.txt`). 3. **Asset Archive Containers (`.bsa` / `.ba2`)**: - Legacy BSA: Folder record tables, file record tables, name hashes, and Zlib-compressed data blocks. - Modern BA2: File type signatures (General vs GNRL vs DX10 texture streams), string table headers, and compression headers. 4. **Compiled Papyrus Script Bytecode (`.pex`)**: Custom object-oriented virtual machine executing compiled `.pex` scripts attached to quests, items, scenes, and magic effects. Scripts define classes, properties, variables, state machines, and event handlers. 5. **NIF Geometry & Animation Linking**: 3D scene graphs (`.nif`) referencing skeleton bones, shader textures, collision hulls, and animation controllers. REA requires a dedicated engine to parse plugin record hierarchies, resolve FormID cross-references across multi-file load orders, disassemble Papyrus `.pex` bytecode, and inspect `.bsa`/`.ba2` archives. ### Proposed Architecture ```mermaid flowchart TD A["Plugin File (.esm/.esp/.esl) / Archive (.bsa/.ba2) / Script (.pex)"] --> B["Plugin & Modular Asset Engine"] B --> C["Archive Container Unpacker (BSA / BA2)"] B --> D["Record & FormID Hierarchy Parser"] B --> E["Papyrus Bytecode Decompiler (.pex)"] C --> C1["BSA Directory & Hash Tables (Zlib Slices)"] C --> C2["BA2 General & DX10 Chunk Deserializer"] D --> D1["GRUP & Record Header Parsing (4-byte Signatures)"] D --> D2["Subrecord Decoding (EDID, FULL, DATA, OBND)"] D --> D3["FormID Reference Graph & Load Order Resolution"] E --> E1[".pex Header, String Table & User Flags"] E --> E2["Class Definitions, Properties & State Machines"] E --> E3["Opcode Stream Disassembly & Event Function Control Flow"] C & D & E --> F["Synthesized Plugin Dependency Tree, Reconstructed Scripts & Asset Manifest"] ``` ### Technical Capabilities & Modules 1. **BSA and BA2 Archive Extraction:** - Parse legacy BSA headers (magic `0x00041534`), folder/file tables, filename hashes, and compression flags. - Parse modern BA2 headers (magic `BTDX`), separate General archives from DX10 texture stream archives, and extract compressed asset files. 2. **Hierarchical Record Parsing (`.esm`, `.esp`, `.esl`):** - Traverse `GRUP` containers, handling group types: top-level record type, in-world children, interior sub-cells, and topic children. - Parse record headers: FormID, flags (compressed, master, deleted), data version, and revision numbers. - Decompress Zlib-compressed records and parse subrecords (4-character tags + 16-bit/32-bit length fields). 3. **FormID Cross-Reference & Conflict Resolution:** - Index all defined FormIDs and build a reference table mapping which records reference or override others. - Resolve relative FormID indexing against master dependencies declared in the `TES4` header (`MAST` subrecords). - Detect record override chains (e.g. Weapon balance tweaks or dialogue overrides). 4. **Papyrus Bytecode Disassembly & Decompilation (`.pex`):** - Parse `.pex` binary structure: magic (`0xFA57C0DE`), compiler version, compilation timestamp, source file name, user name, machine name. - Decode string tables and type tables. - Reconstruct class definitions, parent inheritance, declared properties, and variable types. - Disassemble Papyrus opcodes (`callmethod`, `callparent`, `assign`, `return`, `cast`, `jumpt`, `jumpf`) and decompile state machine transitions into readable Papyrus source code. 5. **Dialogue & Quest Flow Reconstruction:** - Parse `QUST` (Quest) and `DIAL` / `INFO` (Dialogue) records. - Reconstruct quest stage state machines, objective log entries, and dialogue choice branches. ### Acceptance Criteria - [ ] Parsing and safe extraction of assets from both `.bsa` and `.ba2` archives. - [ ] Complete traversal of `GRUP`, record, and subrecord hierarchies across `.esm`, `.esp`, and `.esl` plugins. - [ ] Dynamic FormID mapping linking records to their parent master files. - [ ] Disassembly and source code decompilation of compiled `.pex` Papyrus script bytecodes. - [ ] Generation of an interactive quest and dialogue flowchart. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3624 (Native C++ semantic reconstruction: RTTI, vtables) - Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

bsakaguchi referenced this issue

2026-10-17 21:11:15 +00:00

[PR #1417] [CLOSED] fix(install): print the Linux user prefix in the install plan #2870

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1417

No description provided.

Rows
Columns