[GH-ISSUE #3699] [Feature] Lua and LuaJIT bytecode reverse engineering: chunks, prototypes, fastcall opcodes, CFG and source reconstruction #1408

New issue

Open

opened 2026-10-16 21:05:12 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:12 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3699

Problem Statement & Context

Lua (versions 5.1, 5.2, 5.3, 5.4) and LuaJIT (2.0, 2.1) represent the most pervasive embedded scripting runtime family in interactive applications, game engines, embedded devices, network tools, and desktop plugins.

Compiled Lua files exist in two major binary representations:

Standard Lua Chunks (\x1bLua): Header flags (version, format 0, endianness, int/size_t/Instruction widths, Lua number format), proto hierarchies, instruction arrays, constants (nil, boolean, number, string), upvalue descriptions, local variable scopes, and line-number debugging tables.
LuaJIT 2.0/2.1 Bytecode Chunks (\x1bLJ): Highly optimized binary format featuring variable-length ULEB128 encodings, custom 32-bit register-based opcodes, fastcall built-in opcodes, NaN-tagging 64-bit constant representations, complex prototype flag masks (JIT_F_*), and stripped debug information.

Currently, reverse engineers analyzing compiled scripts must toggle between incompatible external scripts (e.g., unluac, luadec, ljd) that frequently fail on modified headers, custom opcodes, or modern LuaJIT 2.1 GC64 variants. REA needs a native, resilient Lua and LuaJIT bytecode decompiler and control flow reconstructor.

Proposed Architecture

flowchart TD
    A["Compiled Binary Chunk (\x1bLua / \x1bLJ / .luac / .lub)"] --> B["Lua / LuaJIT Semantic Engine"]
    B --> C["Header & Version Detector"]
    B --> D["Prototype & Instruction Disassembler"]
    B --> E["Control Flow & AST Decompiler"]
    
    C --> C1["Lua 5.1 - 5.4 Signature Matrix"]
    C --> C2["LuaJIT 2.0 / 2.1 (v1, v2, GC64) Modes"]
    C --> C3["Endianness & Word-Size Normalization"]
    
    D --> D1["Proto Hierarchy (Outer Function -> Closures)"]
    D --> D2["Opcode Decoding (Standard Opcodes + Fastcalls)"]
    D --> D3["Constants, Upvalues & Environment Resolution"]
    
    E --> E1["Basic Block Construction & Branch Analysis"]
    E --> E2["Loop Detection (ForPrep/ForLoop, While, Repeat)"]
    E --> E3["Synthesized Idiomatic Lua Source Code"]
    
    C & D & E --> F["Clean Reconstructed Lua Source & Prototype Graph"]

Technical Capabilities & Modules

Header Identification & Variant Fingerprinting:
- Detect standard Lua magic (\x1bLua) across version bytes: 0x51 (5.1), 0x52 (5.2), 0x53 (5.3), 0x54 (5.4).
- Detect LuaJIT magic (\x1bLJ) across bytecode versions 1 (LuaJIT 2.0), 2 (LuaJIT 2.1 32-bit), and GC64 (LuaJIT 2.1 64-bit).
- Support common bytecode variations: non-standard instruction sizes, inverted endianness, or custom opcode permutations.
Prototype & Instruction Disassembly:
- Parse function prototype trees recursively: number of parameters, is_vararg flags, maximum stack size.
- Decode register allocations, constant table entries, and upvalue bindings (_ENV vs closure captures).
- Handle LuaJIT fastcall builtins (FNEW, TDUP, TNEW, UCLO, ITERN).
Control Flow Graph (CFG) Reconstruction:
- Partition instruction sequences into basic blocks demarcated by conditional jumps (EQ, LT, LE, TEST, TESTSET) and unconditional jumps (JMP).
- Reconstruct structured loops: numeric for (FORPREP / FORLOOP), generic iterator for (TFORLOOP), pre-tested while, and post-tested repeat-until.
Expression & Statement Synthesis:
- Perform data-flow and live-variable analysis on virtual register assignments.
- De-sugar boolean operations (and / or short-circuit evaluation chains).
- Reconstruct table constructors, multidimensional array indexing, and method invocations (: syntax).
Stripped & Obfuscated Bytecode Resilience:
- Gracefully recover scripts lacking local variable names and line debug records.
- Provide fallback high-level assembly disassembly when AST control-flow structures cannot be unambiguously desugared.

Acceptance Criteria

Successful detection and parsing of standard Lua 5.1 through 5.4 and LuaJIT 2.0 / 2.1 (including GC64) chunks.
Accurate disassembly of all instruction opcodes, constant tables, and upvalue mappings.
Structural recovery of basic blocks, conditional branches, loops, and function closures.
Emission of readable, syntactically valid Lua source code.
Integrated export of the prototype hierarchy tree showing parameter counts, registers, and closures.

Complements #3692 (Roblox/Luau reverse engineering: Luau bytecode, remotes)
Complements #2383 (Audit analysis and reverse engineering capabilities across romhacking)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3699 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context Lua (versions 5.1, 5.2, 5.3, 5.4) and LuaJIT (2.0, 2.1) represent the most pervasive embedded scripting runtime family in interactive applications, game engines, embedded devices, network tools, and desktop plugins. Compiled Lua files exist in two major binary representations: 1. **Standard Lua Chunks (`\x1bLua`)**: Header flags (version, format 0, endianness, int/size_t/Instruction widths, Lua number format), proto hierarchies, instruction arrays, constants (nil, boolean, number, string), upvalue descriptions, local variable scopes, and line-number debugging tables. 2. **LuaJIT 2.0/2.1 Bytecode Chunks (`\x1bLJ`)**: Highly optimized binary format featuring variable-length ULEB128 encodings, custom 32-bit register-based opcodes, fastcall built-in opcodes, NaN-tagging 64-bit constant representations, complex prototype flag masks (`JIT_F_*`), and stripped debug information. Currently, reverse engineers analyzing compiled scripts must toggle between incompatible external scripts (e.g., unluac, luadec, ljd) that frequently fail on modified headers, custom opcodes, or modern LuaJIT 2.1 GC64 variants. REA needs a native, resilient Lua and LuaJIT bytecode decompiler and control flow reconstructor. ### Proposed Architecture ```mermaid flowchart TD A["Compiled Binary Chunk (\x1bLua / \x1bLJ / .luac / .lub)"] --> B["Lua / LuaJIT Semantic Engine"] B --> C["Header & Version Detector"] B --> D["Prototype & Instruction Disassembler"] B --> E["Control Flow & AST Decompiler"] C --> C1["Lua 5.1 - 5.4 Signature Matrix"] C --> C2["LuaJIT 2.0 / 2.1 (v1, v2, GC64) Modes"] C --> C3["Endianness & Word-Size Normalization"] D --> D1["Proto Hierarchy (Outer Function -> Closures)"] D --> D2["Opcode Decoding (Standard Opcodes + Fastcalls)"] D --> D3["Constants, Upvalues & Environment Resolution"] E --> E1["Basic Block Construction & Branch Analysis"] E --> E2["Loop Detection (ForPrep/ForLoop, While, Repeat)"] E --> E3["Synthesized Idiomatic Lua Source Code"] C & D & E --> F["Clean Reconstructed Lua Source & Prototype Graph"] ``` ### Technical Capabilities & Modules 1. **Header Identification & Variant Fingerprinting:** - Detect standard Lua magic (`\x1bLua`) across version bytes: `0x51` (5.1), `0x52` (5.2), `0x53` (5.3), `0x54` (5.4). - Detect LuaJIT magic (`\x1bLJ`) across bytecode versions 1 (LuaJIT 2.0), 2 (LuaJIT 2.1 32-bit), and GC64 (LuaJIT 2.1 64-bit). - Support common bytecode variations: non-standard instruction sizes, inverted endianness, or custom opcode permutations. 2. **Prototype & Instruction Disassembly:** - Parse function prototype trees recursively: number of parameters, `is_vararg` flags, maximum stack size. - Decode register allocations, constant table entries, and upvalue bindings (`_ENV` vs closure captures). - Handle LuaJIT fastcall builtins (`FNEW`, `TDUP`, `TNEW`, `UCLO`, `ITERN`). 3. **Control Flow Graph (CFG) Reconstruction:** - Partition instruction sequences into basic blocks demarcated by conditional jumps (`EQ`, `LT`, `LE`, `TEST`, `TESTSET`) and unconditional jumps (`JMP`). - Reconstruct structured loops: numeric `for` (`FORPREP` / `FORLOOP`), generic iterator `for` (`TFORLOOP`), pre-tested `while`, and post-tested `repeat-until`. 4. **Expression & Statement Synthesis:** - Perform data-flow and live-variable analysis on virtual register assignments. - De-sugar boolean operations (`and` / `or` short-circuit evaluation chains). - Reconstruct table constructors, multidimensional array indexing, and method invocations (`:` syntax). 5. **Stripped & Obfuscated Bytecode Resilience:** - Gracefully recover scripts lacking local variable names and line debug records. - Provide fallback high-level assembly disassembly when AST control-flow structures cannot be unambiguously desugared. ### Acceptance Criteria - [ ] Successful detection and parsing of standard Lua 5.1 through 5.4 and LuaJIT 2.0 / 2.1 (including GC64) chunks. - [ ] Accurate disassembly of all instruction opcodes, constant tables, and upvalue mappings. - [ ] Structural recovery of basic blocks, conditional branches, loops, and function closures. - [ ] Emission of readable, syntactically valid Lua source code. - [ ] Integrated export of the prototype hierarchy tree showing parameter counts, registers, and closures. ### Related Issues & Dependencies - Complements #3692 (Roblox/Luau reverse engineering: Luau bytecode, remotes) - Complements #2383 (Audit analysis and reverse engineering capabilities across romhacking)

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:11:09 +00:00

[PR #1407] [MERGED] fix(javascript)!: stop copying literals into semantic metadata #2863

bsakaguchi referenced this issue

2026-10-17 21:11:09 +00:00

[PR #1408] [MERGED] fix(javascript): retain shorter literal metadata encodings #2864