VTX meshes and KV3 object graphs #1415

New issue

Open

opened 2026-10-16 21:05:15 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:15 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3705

Problem Statement & Context

Valve's Source and Source 2 engine families power dozens of the world's most widely played first-person shooters, multiplayer action games, and modular modifications. Assets across both engine generations are structured into standardized binary packages and lump-based formats:

VPK Virtual Packages (.vpk): Multi-part archives (_dir.vpk index containing directory trees, extensions, checksums, and external data chunk references _000.vpk–_NNN.vpk).
Binary Space Partitioning Maps (.bsp): Lump-based spatial packages containing planes, leaf nodes, brush geometry, visibility matrices (PVS), ambient lighting (lightmaps), embedded packfiles (internal zip lump), and textual entity lump dictionaries (hammerid, classname, target/output triggers).
Material & Texture Systems (.vmt, .vtf): Valve Material Types (declarative KeyValues text files declaring shaders, fallback parameters, and proxy animation blocks) and Valve Texture Format (.vtf binary headers, mipmaps, cubemaps, DXT/BGRA compression).
Skeletal Meshes & Models (.mdl, .vvd, .vtx): Model headers, vertex data lumps (.vvd), optimized vertex index strips (.vtx), collision physics hulls (.phy), and attachment bones.
Modern Resource Formats & KeyValues 3 (KV3): Source 2 compiled resources (_c extensions like .vmdl_c, .vmat_c, .vwrld_c) packing serialized KV3 binary blocks, NTRO introspected structs, and chunked dependency trees.

REA lacks native tooling to unpack multi-part VPK packages, parse BSP entity lump graphs, and deserialize KV3 binary resources.

Proposed Architecture

flowchart TD
    A["Package / Map File (.vpk, .bsp, *_c compiled)"] --> B["VPK & BSP Semantic Engine"]
    B --> C["VPK Archive Slicer (v1 / v2 / multi-part)"]
    B --> D["BSP Lump Parser & Spatial Entity Extractor"]
    B --> E["Material, Model & KV3 Deserializer"]
    
    C --> C1["_dir.vpk Tree Traversal (Ext -> Path -> File)"]
    C --> C2["Chunk Slicing (_000.vpk ... _NNN.vpk)"]
    
    D --> D1["Lump Header (Entities, Textures, Planes, Pakfile)"]
    D --> D2["Entity Lump Parsing (classname, targetname, outputs)"]
    D --> D3["Embedded Pakfile (Internal Zip) Extraction"]
    
    E --> E1[".vmt KeyValues Parsing & .vtf Texture Decoding"]
    E --> E2[".mdl / .vvd / .vtx Mesh & Attachment Bones"]
    E --> E3["KV3 / NTRO Binary Resource Deserialization (*_c)"]
    
    C & D & E --> F["Unified Asset Manifest, Entity Trigger Flowchart & Resource Topology"]

Technical Capabilities & Modules

VPK Archive Ingestion (v1 and v2):
- Parse VPK header signatures (0x55AA1234), header versions (v1, v2), and directory tree tables.
- Resolve multi-chunk archive dependencies linking master index files (*_dir.vpk) to physical payload chunks.
- Extract raw assets with CRC32 integrity validation.
BSP Map Lump Traversal & Entity Extraction:
- Parse BSP version headers and the 64-entry lump header table.
- Extract Lump 0 (Entities): parse key-value text pairs, identifying entity classes (info_player_start, trigger_multiple, func_door), logic relays, and target/output connections (OnTrigger -> target -> action).
- Extract embedded zip lumps (custom textures, sound scripts, models packed inside the map).
Material & Texture Decoding:
- Parse .vmt files (KeyValues 1 syntax): resolve base textures, bump maps, envmaps, and texture transform proxies.
- Decode .vtf binary headers: extract mipmaps, resolve texture format flags (DXT1, DXT5, BGR888, BGRA8888), and export to PNG.
Model & Geometry Inspection (.mdl):
- Parse .mdl headers: bone hierarchies, animation sequences, body parts, and skin families.
- Correlate with vertex buffers (.vvd) and strip models (.vtx).
Source 2 Compiled Resource Deserialization (*_c):
- Parse modern compiled resource headers (ResourceExt_t, block count, block descriptors).
- Deserialize KeyValues 3 (KV3) text and binary formats (LZ4 / Zstd compressed).
- Resolve resource dependency lists (REDIS / DATA blocks) to map inter-asset references.

Acceptance Criteria

Full extraction of single-file and multi-chunk .vpk packages.
Parsing of .bsp map headers and extraction of entity relationship/output graphs.
Extraction of packed assets from embedded map pakfile lumps.
Parsing of .vmt material scripts and conversion of .vtf textures to PNG.
Deserialization of Source 2 compiled binary resource containers (*_c) and extraction of KV3 metadata.

Complements #3621 (Unknown binary file-format inference)
Complements #3620 (Shader and GPU Bytecode Reverse Engineering)
Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3705 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context Valve's Source and Source 2 engine families power dozens of the world's most widely played first-person shooters, multiplayer action games, and modular modifications. Assets across both engine generations are structured into standardized binary packages and lump-based formats: 1. **VPK Virtual Packages (`.vpk`)**: Multi-part archives (`_dir.vpk` index containing directory trees, extensions, checksums, and external data chunk references `_000.vpk`–`_NNN.vpk`). 2. **Binary Space Partitioning Maps (`.bsp`)**: Lump-based spatial packages containing planes, leaf nodes, brush geometry, visibility matrices (PVS), ambient lighting (lightmaps), embedded packfiles (internal zip lump), and textual entity lump dictionaries (`hammerid`, `classname`, target/output triggers). 3. **Material & Texture Systems (`.vmt`, `.vtf`)**: Valve Material Types (declarative KeyValues text files declaring shaders, fallback parameters, and proxy animation blocks) and Valve Texture Format (`.vtf` binary headers, mipmaps, cubemaps, DXT/BGRA compression). 4. **Skeletal Meshes & Models (`.mdl`, `.vvd`, `.vtx`)**: Model headers, vertex data lumps (`.vvd`), optimized vertex index strips (`.vtx`), collision physics hulls (`.phy`), and attachment bones. 5. **Modern Resource Formats & KeyValues 3 (KV3)**: Source 2 compiled resources (`_c` extensions like `.vmdl_c`, `.vmat_c`, `.vwrld_c`) packing serialized KV3 binary blocks, NTRO introspected structs, and chunked dependency trees. REA lacks native tooling to unpack multi-part VPK packages, parse BSP entity lump graphs, and deserialize KV3 binary resources. ### Proposed Architecture ```mermaid flowchart TD A["Package / Map File (.vpk, .bsp, *_c compiled)"] --> B["VPK & BSP Semantic Engine"] B --> C["VPK Archive Slicer (v1 / v2 / multi-part)"] B --> D["BSP Lump Parser & Spatial Entity Extractor"] B --> E["Material, Model & KV3 Deserializer"] C --> C1["_dir.vpk Tree Traversal (Ext -> Path -> File)"] C --> C2["Chunk Slicing (_000.vpk ... _NNN.vpk)"] D --> D1["Lump Header (Entities, Textures, Planes, Pakfile)"] D --> D2["Entity Lump Parsing (classname, targetname, outputs)"] D --> D3["Embedded Pakfile (Internal Zip) Extraction"] E --> E1[".vmt KeyValues Parsing & .vtf Texture Decoding"] E --> E2[".mdl / .vvd / .vtx Mesh & Attachment Bones"] E --> E3["KV3 / NTRO Binary Resource Deserialization (*_c)"] C & D & E --> F["Unified Asset Manifest, Entity Trigger Flowchart & Resource Topology"] ``` ### Technical Capabilities & Modules 1. **VPK Archive Ingestion (v1 and v2):** - Parse VPK header signatures (`0x55AA1234`), header versions (v1, v2), and directory tree tables. - Resolve multi-chunk archive dependencies linking master index files (`*_dir.vpk`) to physical payload chunks. - Extract raw assets with CRC32 integrity validation. 2. **BSP Map Lump Traversal & Entity Extraction:** - Parse BSP version headers and the 64-entry lump header table. - Extract Lump 0 (Entities): parse key-value text pairs, identifying entity classes (`info_player_start`, `trigger_multiple`, `func_door`), logic relays, and target/output connections (`OnTrigger` -> `target` -> `action`). - Extract embedded zip lumps (custom textures, sound scripts, models packed inside the map). 3. **Material & Texture Decoding:** - Parse `.vmt` files (KeyValues 1 syntax): resolve base textures, bump maps, envmaps, and texture transform proxies. - Decode `.vtf` binary headers: extract mipmaps, resolve texture format flags (DXT1, DXT5, BGR888, BGRA8888), and export to PNG. 4. **Model & Geometry Inspection (`.mdl`):** - Parse `.mdl` headers: bone hierarchies, animation sequences, body parts, and skin families. - Correlate with vertex buffers (`.vvd`) and strip models (`.vtx`). 5. **Source 2 Compiled Resource Deserialization (`*_c`):** - Parse modern compiled resource headers (`ResourceExt_t`, block count, block descriptors). - Deserialize KeyValues 3 (KV3) text and binary formats (LZ4 / Zstd compressed). - Resolve resource dependency lists (`REDIS` / `DATA` blocks) to map inter-asset references. ### Acceptance Criteria - [ ] Full extraction of single-file and multi-chunk `.vpk` packages. - [ ] Parsing of `.bsp` map headers and extraction of entity relationship/output graphs. - [ ] Extraction of packed assets from embedded map pakfile lumps. - [ ] Parsing of `.vmt` material scripts and conversion of `.vtf` textures to PNG. - [ ] Deserialization of Source 2 compiled binary resource containers (`*_c`) and extraction of KV3 metadata. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3620 (Shader and GPU Bytecode Reverse Engineering) - Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

bsakaguchi referenced this issue

2026-10-17 21:11:12 +00:00

[PR #1415] [MERGED] feat(clients): register Qwen Code during setup #2869

bsakaguchi referenced this issue

2026-10-17 21:11:58 +00:00

[PR #1511] [MERGED] test(clients): verify native Qwen Pi and Hermes workflows #2931

bsakaguchi referenced this issue

2026-10-17 21:12:25 +00:00

[PR #1564] [MERGED] feat(clients): register ZCode during setup #2974

bsakaguchi referenced this issue

2026-10-17 21:12:31 +00:00

[PR #1572] [MERGED] feat(clients): register Qoder CLI during setup #2980