[GH-ISSUE #3712] [Feature] HKX binary packfile and tagfile reverse engineering: reflection metadata, skeleton hierarchies, ragdoll constraints and behavior animation graphs #1423

New issue

Open

opened 2026-10-16 21:05:19 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:19 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3712

Problem Statement & Context

The Havok physics, ragdoll, animation, and behavior middleware ecosystem is embedded in hundreds of industry-leading AAA video games, simulations, and virtual worlds. Assets are distributed in proprietary binary packfiles and tagfiles (.hkx):

HKX Container Formats:
- Binary Packfile: Platform-specific header (hk_2010, hk_2012, hk_2014, hk_2016), pointer size (32-bit vs 64-bit), little vs big endian, padding flags, and distinct data sections: __header__, __types__, __data__.
- Tagfile / TagXML: Modern tag-based format storing serialized object instances with self-contained type registries.
Class Reflection & Serialization Registry: Hundreds of reflected C++ types (hkbBehaviorGraph, hkbStateMachine, hkaSkeleton, hkaAnimationBinding, hkpRigidBody, hkpBoxShape, hkpConstraintInstance).
Behavior State Machines (hkbStateMachine, hkbClipGenerator): Hierarchical state machines defining animation states, transitions, blend trees, variable bindings, and event triggers (hkbEvent).
Skeletal Rigging & Animation (hkaSkeleton, hkaSplineCompressedAnimation): Bone transform hierarchies, reference poses, floating-point spline compression, and skinning binding indices.
Rigid Body Physics & Constraints (hkpRigidBody, hkpConstraintData): Collision shape hierarchies (box, sphere, capsule, convex vertices, MOPP trees), ragdoll joint limits (hinge, ball-and-socket, prismatic), and mass properties.

REA currently lacks a dedicated HKX reader and behavior state graph reconstructor.

Proposed Architecture

flowchart TD
    A["Havok File (.hkx packfile / tagfile)"] --> B["HKX Semantic Engine"]
    B --> C["HKX Header & Section Parser"]
    B --> D["Havok Reflection & Type Registry"]
    B --> E["Behavior, Physics & Skeletal Reconstructors"]
    
    C --> C1["Magic Verification (Packfile / Tagfile)"]
    C --> C2["Section Table (__header__, __types__, __data__)"]
    C --> C3["Pointer Size (32/64-bit) & Endianness Fixup"]
    
    D --> D1["Type Signature & Class Descriptor Deserialization"]
    D --> D2["Virtual Pointer Relocation Table Processing"]
    
    E --> E1["hkbBehaviorGraph State Machine & Blend Tree Reconstructor"]
    E --> E2["hkaSkeleton Bone Hierarchy & Reference Pose Extractor"]
    E --> E3["hkpRigidBody & Collision Hull Deserializer"]
    
    C & D & E --> F["Synthesized Animation State Flowchart, Skeletal Tree & Physics Blueprint"]

Technical Capabilities & Modules

Packfile and Tagfile Header Parsing:
- Detect binary packfile magic (0x57E0E057) and tagfile signatures across Havok version milestones (2010.x through 2020.x).
- Process pointer relocation tables linking __data__ object instances with __types__ class descriptors.
- Handle cross-platform endianness and pointer-width normalization.
Reflection & Type Registry Engine:
- Parse embedded class descriptors (hkClass, hkClassMember, hkClassEnum).
- Reconstruct strongly-typed C++ struct instances and member values.
- Support reflected types across Havok modules: Physics (hkp), Animation (hka), Behavior (hkb), AI (hkai), and Cloth (hcl).
Behavior Graph & State Machine Reconstruction:
- Reconstruct hkbStateMachine hierarchies: nested states, transition rules, transition conditions, and event dispatchers.
- Trace hkbBlendTree, hkbClipGenerator, and hkbModifier pipeline flow.
- Generate interactive Mermaid flowcharts depicting character animation state machines and event transitions.
Skeleton & Animation Inspection:
- Extract hkaSkeleton hierarchies: bone names, parent indices, local transforms, and reference poses.
- Parse hkaAnimationBinding tracks and decompress spline-compressed animation curves.
Collision & Physics Shape Analysis:
- Deserialize hkpRigidBody parameters: mass, inertia tensor, friction, restitution, and collision filtering bitmasks.
- Decode basic shapes (sphere, box, capsule) and convex vertex hulls.

Acceptance Criteria

Support static parsing of binary .hkx packfiles and tagfiles across 32-bit and 64-bit versions.
Successful relocation and object instantiation using Havok reflection metadata.
Generation of an interactive state transition diagram for hkbBehaviorGraph state machines.
Extraction of skeletal bone trees and reference transforms from hkaSkeleton.
Export of the complete HKX object graph into human-readable JSON representations.

Complements #3621 (Unknown binary file-format inference)
Complements #3624 (Native C++ semantic reconstruction: RTTI, vtables)
Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3712 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context The Havok physics, ragdoll, animation, and behavior middleware ecosystem is embedded in hundreds of industry-leading AAA video games, simulations, and virtual worlds. Assets are distributed in proprietary binary packfiles and tagfiles (`.hkx`): 1. **HKX Container Formats**: - Binary Packfile: Platform-specific header (`hk_2010`, `hk_2012`, `hk_2014`, `hk_2016`), pointer size (32-bit vs 64-bit), little vs big endian, padding flags, and distinct data sections: `__header__`, `__types__`, `__data__`. - Tagfile / TagXML: Modern tag-based format storing serialized object instances with self-contained type registries. 2. **Class Reflection & Serialization Registry**: Hundreds of reflected C++ types (`hkbBehaviorGraph`, `hkbStateMachine`, `hkaSkeleton`, `hkaAnimationBinding`, `hkpRigidBody`, `hkpBoxShape`, `hkpConstraintInstance`). 3. **Behavior State Machines (`hkbStateMachine`, `hkbClipGenerator`)**: Hierarchical state machines defining animation states, transitions, blend trees, variable bindings, and event triggers (`hkbEvent`). 4. **Skeletal Rigging & Animation (`hkaSkeleton`, `hkaSplineCompressedAnimation`)**: Bone transform hierarchies, reference poses, floating-point spline compression, and skinning binding indices. 5. **Rigid Body Physics & Constraints (`hkpRigidBody`, `hkpConstraintData`)**: Collision shape hierarchies (box, sphere, capsule, convex vertices, MOPP trees), ragdoll joint limits (hinge, ball-and-socket, prismatic), and mass properties. REA currently lacks a dedicated HKX reader and behavior state graph reconstructor. ### Proposed Architecture ```mermaid flowchart TD A["Havok File (.hkx packfile / tagfile)"] --> B["HKX Semantic Engine"] B --> C["HKX Header & Section Parser"] B --> D["Havok Reflection & Type Registry"] B --> E["Behavior, Physics & Skeletal Reconstructors"] C --> C1["Magic Verification (Packfile / Tagfile)"] C --> C2["Section Table (__header__, __types__, __data__)"] C --> C3["Pointer Size (32/64-bit) & Endianness Fixup"] D --> D1["Type Signature & Class Descriptor Deserialization"] D --> D2["Virtual Pointer Relocation Table Processing"] E --> E1["hkbBehaviorGraph State Machine & Blend Tree Reconstructor"] E --> E2["hkaSkeleton Bone Hierarchy & Reference Pose Extractor"] E --> E3["hkpRigidBody & Collision Hull Deserializer"] C & D & E --> F["Synthesized Animation State Flowchart, Skeletal Tree & Physics Blueprint"] ``` ### Technical Capabilities & Modules 1. **Packfile and Tagfile Header Parsing:** - Detect binary packfile magic (`0x57E0E057`) and tagfile signatures across Havok version milestones (2010.x through 2020.x). - Process pointer relocation tables linking `__data__` object instances with `__types__` class descriptors. - Handle cross-platform endianness and pointer-width normalization. 2. **Reflection & Type Registry Engine:** - Parse embedded class descriptors (`hkClass`, `hkClassMember`, `hkClassEnum`). - Reconstruct strongly-typed C++ struct instances and member values. - Support reflected types across Havok modules: Physics (`hkp`), Animation (`hka`), Behavior (`hkb`), AI (`hkai`), and Cloth (`hcl`). 3. **Behavior Graph & State Machine Reconstruction:** - Reconstruct `hkbStateMachine` hierarchies: nested states, transition rules, transition conditions, and event dispatchers. - Trace `hkbBlendTree`, `hkbClipGenerator`, and `hkbModifier` pipeline flow. - Generate interactive Mermaid flowcharts depicting character animation state machines and event transitions. 4. **Skeleton & Animation Inspection:** - Extract `hkaSkeleton` hierarchies: bone names, parent indices, local transforms, and reference poses. - Parse `hkaAnimationBinding` tracks and decompress spline-compressed animation curves. 5. **Collision & Physics Shape Analysis:** - Deserialize `hkpRigidBody` parameters: mass, inertia tensor, friction, restitution, and collision filtering bitmasks. - Decode basic shapes (sphere, box, capsule) and convex vertex hulls. ### Acceptance Criteria - [ ] Support static parsing of binary `.hkx` packfiles and tagfiles across 32-bit and 64-bit versions. - [ ] Successful relocation and object instantiation using Havok reflection metadata. - [ ] Generation of an interactive state transition diagram for `hkbBehaviorGraph` state machines. - [ ] Extraction of skeletal bone trees and reference transforms from `hkaSkeleton`. - [ ] Export of the complete HKX object graph into human-readable JSON representations. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3624 (Native C++ semantic reconstruction: RTTI, vtables) - Complements #3698 (Interactive entertainment runtime and asset reverse engineering tracker)

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:11:18 +00:00

[PR #1423] [MERGED] fix(javascript)!: avoid endpoint payload copies in graph metadata #2875

bsakaguchi referenced this issue

2026-10-17 21:11:20 +00:00

[PR #1427] [MERGED] fix(javascript)!: avoid event payload copies in graph metadata #2879