[GH-ISSUE #3689] [Feature] NSIS reverse engineering: installer payloads, script instruction stream, sections/functions, plugins and install-action reconstruction #1399

New issue

Open

opened 2026-10-16 21:05:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:06 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3689

Problem Statement & Context

NSIS (Nullsoft Scriptable Install System) is an industry-standard, open-source script-driven installer engine used across thousands of Windows applications (e.g., VLC Media Player, Winamp, Notepad++, Mozilla Firefox installers).

Unlike declarative installers, an NSIS executable (installer.exe) is an interpreter runtime driven by a compiled, low-level bytecode instruction stream. Reversing an NSIS installer requires analyzing:

Archive Container & Compression: Solid or non-solid LZMA, Bzip2, or Zlib compressed blocks located at the PE EOF overlay.
Compiled Bytecode Instruction Stream: Sequential array of opcodes (EW_ASSIGN, EW_PUSHPOP, EW_EXEC, EW_WRITEINI, EW_WRITEREG, EW_CREATEDIR, EW_EXTRACTFILE).
Sections & Functions: User-selectable installation sections, hidden sections, and global functions (e.g. .onInit, .onInstSuccess, un.onInit).
Strings & Variable Pools: String block decoding, parameter substitutions ($INSTDIR, $SYSDIR, $TEMP), and numbered user variables ($0 to $9, $R0 to $R9).
Plugins System: Custom 32-bit/64-bit DLLs extracted dynamically to $PLUGINSDIR and invoked via CallInstDLL for networking, XML parsing, or elevation.

Without a dedicated NSIS decompiler and static behavior reconstruction engine, analysts must rely on brittle unpackers that extract files but discard the installer's control flow, registry actions, and execution triggers.

Proposed Architecture

flowchart TD
    A["NSIS Installer Executable (installer.exe)"] --> B["NSIS Semantic Engine"]
    B --> C["NSIS Overlay & Container Unpacker"]
    B --> D["Opcode Stream Disassembler & Decompiler"]
    B --> E["Plugin & External Invocation Analyzer"]
    
    C --> C1["NSIS Header Signature (0xDEADBEEF / Nullsoft)"]
    C --> C2["Decompression (LZMA, Bzip2, Zlib)"]
    C --> C3["Extracted Files & Resources"]
    
    D --> D1["Opcode Disassembly (EW_EXEC, EW_WRITEREG, EW_EXTRACTFILE)"]
    D --> D2["Variable & String Block Substitution ($INSTDIR, $0)"]
    D --> D3["Control Flow (Goto, IfErrors, IntCmp, Abort)"]
    D --> D4["Section & Function Structure (.onInit, un.onInit)"]
    
    E --> E1["$PLUGINSDIR DLL Extractions"]
    E --> E2["CallInstDLL Native Interfacing"]
    
    C & D & E --> F["Decompiled NSIS Script (.nsi) & Installation Action Graph"]

Technical Capabilities & Modules

Header Identification & Stream Extraction:
- Scan PE overlay for NSIS magic signature (0xDEADBEEF followed by NullsoftInst or NullsoftScriptableInstallSystem).
- Identify header flags, solid compression state, and compression algorithm (LZMA, Bzip2, Zlib).
- Extract raw blocks: entries (opcodes), strings, language tables, and embedded data streams.
Bytecode Disassembly & Variable Resolution:
- Disassemble NSIS opcodes (entry structures) into structured instructions.
- Resolve internal variables ($0–$9, $R0–$R9, $INSTDIR, $EXEDIR, $PROGRAMFILES).
- Track variable assignments and stack operations (Push, Pop, StrCpy).
Control Flow & Script Decompilation:
- Reconstruct control flow graphs from relative opcode jump offsets (Goto, IfErrors, IfFileExists, IntCmp, StrCmp).
- Demangle user functions and callback functions: .onInit, .onInstSuccess, .onGUIInit, and uninstaller functions (un.*).
- Synthesize a readable .nsi script format matching human-authored NSIS syntax.
Installation Action Modeling:
- Reconstruct filesystem modifications: File, SetOutPath, Delete, RMDir.
- Reconstruct registry actions: WriteRegStr, WriteRegDWORD, DeleteRegKey.
- Trace executable invocations: Exec, ExecWait, ExecShell.
Plugin Call & Native DLL Analysis:
- Identify embedded DLL plugins written to $PLUGINSDIR.
- Map CallInstDLL calls and decode passed arguments and return values.

Acceptance Criteria

Reliable detection and unpacking of NSIS installers across all major NSIS versions (v2.x, v3.x).
Disassembly of NSIS bytecode instruction stream into human-readable instructions.
Decompilation of script logic back into clean, structured .nsi script format.
Full extraction and attribution of payload files, registry modifications, and $PLUGINSDIR DLL calls.
Simulated installation blueprint detailing predicted system changes without executing the binary.

Complements #3621 (Unknown binary file-format inference)
Complements #3626 (Windows kernel driver reverse engineering: INF correlation)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3689 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context NSIS (Nullsoft Scriptable Install System) is an industry-standard, open-source script-driven installer engine used across thousands of Windows applications (e.g., VLC Media Player, Winamp, Notepad++, Mozilla Firefox installers). Unlike declarative installers, an NSIS executable (`installer.exe`) is an interpreter runtime driven by a compiled, low-level bytecode instruction stream. Reversing an NSIS installer requires analyzing: 1. **Archive Container & Compression**: Solid or non-solid LZMA, Bzip2, or Zlib compressed blocks located at the PE EOF overlay. 2. **Compiled Bytecode Instruction Stream**: Sequential array of opcodes (`EW_ASSIGN`, `EW_PUSHPOP`, `EW_EXEC`, `EW_WRITEINI`, `EW_WRITEREG`, `EW_CREATEDIR`, `EW_EXTRACTFILE`). 3. **Sections & Functions**: User-selectable installation sections, hidden sections, and global functions (e.g. `.onInit`, `.onInstSuccess`, `un.onInit`). 4. **Strings & Variable Pools**: String block decoding, parameter substitutions (`$INSTDIR`, `$SYSDIR`, `$TEMP`), and numbered user variables (`$0` to `$9`, `$R0` to `$R9`). 5. **Plugins System**: Custom 32-bit/64-bit DLLs extracted dynamically to `$PLUGINSDIR` and invoked via `CallInstDLL` for networking, XML parsing, or elevation. Without a dedicated NSIS decompiler and static behavior reconstruction engine, analysts must rely on brittle unpackers that extract files but discard the installer's control flow, registry actions, and execution triggers. ### Proposed Architecture ```mermaid flowchart TD A["NSIS Installer Executable (installer.exe)"] --> B["NSIS Semantic Engine"] B --> C["NSIS Overlay & Container Unpacker"] B --> D["Opcode Stream Disassembler & Decompiler"] B --> E["Plugin & External Invocation Analyzer"] C --> C1["NSIS Header Signature (0xDEADBEEF / Nullsoft)"] C --> C2["Decompression (LZMA, Bzip2, Zlib)"] C --> C3["Extracted Files & Resources"] D --> D1["Opcode Disassembly (EW_EXEC, EW_WRITEREG, EW_EXTRACTFILE)"] D --> D2["Variable & String Block Substitution ($INSTDIR, $0)"] D --> D3["Control Flow (Goto, IfErrors, IntCmp, Abort)"] D --> D4["Section & Function Structure (.onInit, un.onInit)"] E --> E1["$PLUGINSDIR DLL Extractions"] E --> E2["CallInstDLL Native Interfacing"] C & D & E --> F["Decompiled NSIS Script (.nsi) & Installation Action Graph"] ``` ### Technical Capabilities & Modules 1. **Header Identification & Stream Extraction:** - Scan PE overlay for NSIS magic signature (`0xDEADBEEF` followed by `NullsoftInst` or `NullsoftScriptableInstallSystem`). - Identify header flags, solid compression state, and compression algorithm (LZMA, Bzip2, Zlib). - Extract raw blocks: entries (opcodes), strings, language tables, and embedded data streams. 2. **Bytecode Disassembly & Variable Resolution:** - Disassemble NSIS opcodes (`entry` structures) into structured instructions. - Resolve internal variables (`$0`–`$9`, `$R0`–`$R9`, `$INSTDIR`, `$EXEDIR`, `$PROGRAMFILES`). - Track variable assignments and stack operations (`Push`, `Pop`, `StrCpy`). 3. **Control Flow & Script Decompilation:** - Reconstruct control flow graphs from relative opcode jump offsets (`Goto`, `IfErrors`, `IfFileExists`, `IntCmp`, `StrCmp`). - Demangle user functions and callback functions: `.onInit`, `.onInstSuccess`, `.onGUIInit`, and uninstaller functions (`un.*`). - Synthesize a readable `.nsi` script format matching human-authored NSIS syntax. 4. **Installation Action Modeling:** - Reconstruct filesystem modifications: `File`, `SetOutPath`, `Delete`, `RMDir`. - Reconstruct registry actions: `WriteRegStr`, `WriteRegDWORD`, `DeleteRegKey`. - Trace executable invocations: `Exec`, `ExecWait`, `ExecShell`. 5. **Plugin Call & Native DLL Analysis:** - Identify embedded DLL plugins written to `$PLUGINSDIR`. - Map `CallInstDLL` calls and decode passed arguments and return values. ### Acceptance Criteria - [ ] Reliable detection and unpacking of NSIS installers across all major NSIS versions (v2.x, v3.x). - [ ] Disassembly of NSIS bytecode instruction stream into human-readable instructions. - [ ] Decompilation of script logic back into clean, structured `.nsi` script format. - [ ] Full extraction and attribution of payload files, registry modifications, and `$PLUGINSDIR` DLL calls. - [ ] Simulated installation blueprint detailing predicted system changes without executing the binary. ### Related Issues & Dependencies - Complements #3621 (Unknown binary file-format inference) - Complements #3626 (Windows kernel driver reverse engineering: INF correlation)

bsakaguchi referenced this issue

2026-10-16 21:05:45 +00:00

[GH-ISSUE #3795] [Tracker] Open documentation issues and pull requests (October 2026) #1468

bsakaguchi referenced this issue

2026-10-16 21:05:46 +00:00

[GH-ISSUE #3795] [Tracker] Open documentation issues and pull requests (October 2026) #1468

bsakaguchi referenced this issue

2026-10-17 21:11:04 +00:00

[PR #1399] [MERGED] chore(main): release rea-agents 6.4.0 #2855