[GH-ISSUE #3674] [Feature] CUDA reverse engineering: PTX, cubin/fatbin, SASS kernels and host-device call correlation #1384

New issue

Open

opened 2026-10-16 21:04:56 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:56 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3674

Summary

NVIDIA CUDA is the dominant GPU computing platform for deep learning acceleration, scientific simulation, cryptographic mining, and high-performance computing (HPC). CUDA applications bundle GPU device code inside native host executables (PE, ELF) or dynamic libraries (.so, .dll) as:

CUDA Fatbins (.fatbin): Multi-architecture containers wrapping compiled GPU code;
PTX (Parallel Thread Execution): Low-level virtual machine assembly code (intermediate representation);
Cubin (.cubin): Compiled ELF-based native GPU binaries targeting specific Streaming Multiprocessor (SM) architectures (e.g. sm_70 Volta, sm_80 Ampere, sm_89 Ada, sm_90 Hopper, sm_100 Blackwell);
SASS (Streaming Assembler): Native machine code executed directly by GPU hardware threads.

Currently, REA analyzes native host binaries (#3624) and GPU shaders (#3620), but treats CUDA fatbinary sections (.nv_fatbin) as opaque data buffers.

This feature proposal introduces dedicated CUDA Reverse Engineering to REA, enabling automated extraction of CUDA fatbins, parsing of PTX assembly, ELF cubin analysis, SASS kernel disassembly, and correlation of host cudaLaunchKernel calls to device GPU kernels.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Native Executable / Shared Lib   |
                                    |        or Standalone .fatbin / .cubin   |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | CUDA Fatbin Container     |                                   | Host-to-Device Call       |
           | Unpacker (.nv_fatbin)     |                                   | Correlation (cudaLaunch)  |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | PTX Intermediate Assembly Inverter    |
                                     | (.version, .target, .reg, kernel ops) |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Cubin ELF Container Parser|                                   | SASS Machine Code         |
           | (NVVM sections, symbols)  |                                   | Disassembler (Ampere,Hop) |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: GPU Kernel Dossier,     |
                                     | Memory Requirements & Launch Geometry |
                                     +---------------------------------------+

1. CUDA Fatbin Container Extraction

Section Carving: Scan PE .rdata / .data and ELF .nv_fatbin sections for the Fatbinary magic header (0xBA55ED50 / 0x11A0ED41).
Archive Unpacking: Parse fatbin header structures (uncompressed, Zlib, or LZ4 compressed slices):
- Extract target architecture tags (e.g. compute_80, sm_80, compute_90, sm_90).
- Extract embedded PTX text streams and binary ELF .cubin objects.

2. PTX (Parallel Thread Execution) Analysis

PTX Header & Directive Parsing: Parse .version, .target, and address size directives (.address_size 64).
Kernel & Function Extraction:
- Extract .entry kernel declarations: parameter lists, parameter memory spaces (.param), register allocations (.reg).
- Memory space directives: .global, .shared, .const, .local.
Instruction Disassembly: Disassemble PTX vector, arithmetic, memory, and synchronization instructions (ld.global, st.global, mma.sync Tensor Core operations, bar.sync).

3. Cubin ELF Binary & SASS Disassembly

Cubin ELF Structure: Parse GPU ELF headers (EM_CUDA machine type 0xBE):
- Section tables: .nv.info, .nv.text.<kernel_name>, .nv.constant.
- Symbol tables: function symbols, shared memory sizes, register count requirements per thread.
SASS Machine Code Disassembly: Disassemble SASS instructions across major GPU generations:
- FP32/FP16 arithmetic (FADD, FMUL, FFMA, HFMA2).
- Tensor Core Matrix Multiply-Accumulate (HMMA, IMMA, GMMA, WGMMA).
- Shared memory and asynchronous transfer instructions (LDSM, TMA Tensor Memory Accelerator).

4. Host-Device Call Correlation

Runtime API Tracing: Invert host-side CUDA runtime and driver API calls:
- Registration: __cudaRegisterFatBinary, __cudaRegisterFunction.
- Kernel Launches: cudaLaunchKernel, cudaLaunchCooperativeKernel, cuLaunchKernel.
Launch Geometry Mapping: Extract grid dimensions (gridDim), block dimensions (blockDim), shared memory byte size, and parameter buffer bindings connecting host calling code to target GPU kernels.

Acceptance Criteria

Introduce analyze_cuda_binary in REA CLI and MCP contracts, accepting native PE/ELF executables and .fatbin/.cubin files.
Safely extract embedded PTX and cubin slices from synthetic CUDA fatbin test fixtures.
Parse PTX assembly directives, outputting kernel entrypoints, parameters, and register budgets into structured JSON.
Extract cubin ELF sections and identify target SM architecture versions.
Correlate host cudaLaunchKernel invocations with target device kernel symbols.

#3620 [Feature] Shader and GPU Bytecode Reverse Engineering
#3624 [Feature] Native C++ semantic reconstruction
#3637 [Feature] ML model graph reverse engineering

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3674 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Reverse Engineering](https://img.shields.io/badge/domain-reverse_engineering-blueviolet.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary NVIDIA CUDA is the dominant GPU computing platform for deep learning acceleration, scientific simulation, cryptographic mining, and high-performance computing (HPC). CUDA applications bundle GPU device code inside native host executables (PE, ELF) or dynamic libraries (`.so`, `.dll`) as: - **CUDA Fatbins (`.fatbin`):** Multi-architecture containers wrapping compiled GPU code; - **PTX (Parallel Thread Execution):** Low-level virtual machine assembly code (intermediate representation); - **Cubin (`.cubin`):** Compiled ELF-based native GPU binaries targeting specific Streaming Multiprocessor (SM) architectures (e.g. `sm_70` Volta, `sm_80` Ampere, `sm_89` Ada, `sm_90` Hopper, `sm_100` Blackwell); - **SASS (Streaming Assembler):** Native machine code executed directly by GPU hardware threads. Currently, REA analyzes native host binaries (#3624) and GPU shaders (#3620), but treats CUDA fatbinary sections (`.nv_fatbin`) as opaque data buffers. This feature proposal introduces dedicated **CUDA Reverse Engineering** to REA, enabling automated extraction of CUDA fatbins, parsing of PTX assembly, ELF cubin analysis, SASS kernel disassembly, and correlation of host `cudaLaunchKernel` calls to device GPU kernels. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Native Executable / Shared Lib | | or Standalone .fatbin / .cubin | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | CUDA Fatbin Container | | Host-to-Device Call | | Unpacker (.nv_fatbin) | | Correlation (cudaLaunch) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | PTX Intermediate Assembly Inverter | | (.version, .target, .reg, kernel ops) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Cubin ELF Container Parser| | SASS Machine Code | | (NVVM sections, symbols) | | Disassembler (Ampere,Hop) | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: GPU Kernel Dossier, | | Memory Requirements & Launch Geometry | +---------------------------------------+ ``` ### 1. CUDA Fatbin Container Extraction - **Section Carving:** Scan PE `.rdata` / `.data` and ELF `.nv_fatbin` sections for the Fatbinary magic header (`0xBA55ED50` / `0x11A0ED41`). - **Archive Unpacking:** Parse fatbin header structures (uncompressed, Zlib, or LZ4 compressed slices): - Extract target architecture tags (e.g. `compute_80`, `sm_80`, `compute_90`, `sm_90`). - Extract embedded PTX text streams and binary ELF `.cubin` objects. ### 2. PTX (Parallel Thread Execution) Analysis - **PTX Header & Directive Parsing:** Parse `.version`, `.target`, and address size directives (`.address_size 64`). - **Kernel & Function Extraction:** - Extract `.entry` kernel declarations: parameter lists, parameter memory spaces (`.param`), register allocations (`.reg`). - Memory space directives: `.global`, `.shared`, `.const`, `.local`. - **Instruction Disassembly:** Disassemble PTX vector, arithmetic, memory, and synchronization instructions (`ld.global`, `st.global`, `mma.sync` Tensor Core operations, `bar.sync`). ### 3. Cubin ELF Binary & SASS Disassembly - **Cubin ELF Structure:** Parse GPU ELF headers (`EM_CUDA` machine type `0xBE`): - Section tables: `.nv.info`, `.nv.text.<kernel_name>`, `.nv.constant`. - Symbol tables: function symbols, shared memory sizes, register count requirements per thread. - **SASS Machine Code Disassembly:** Disassemble SASS instructions across major GPU generations: - FP32/FP16 arithmetic (`FADD`, `FMUL`, `FFMA`, `HFMA2`). - Tensor Core Matrix Multiply-Accumulate (`HMMA`, `IMMA`, `GMMA`, `WGMMA`). - Shared memory and asynchronous transfer instructions (`LDSM`, `TMA` Tensor Memory Accelerator). ### 4. Host-Device Call Correlation - **Runtime API Tracing:** Invert host-side CUDA runtime and driver API calls: - Registration: `__cudaRegisterFatBinary`, `__cudaRegisterFunction`. - Kernel Launches: `cudaLaunchKernel`, `cudaLaunchCooperativeKernel`, `cuLaunchKernel`. - **Launch Geometry Mapping:** Extract grid dimensions (`gridDim`), block dimensions (`blockDim`), shared memory byte size, and parameter buffer bindings connecting host calling code to target GPU kernels. --- ## Acceptance Criteria 1. Introduce `analyze_cuda_binary` in REA CLI and MCP contracts, accepting native PE/ELF executables and `.fatbin`/`.cubin` files. 2. Safely extract embedded PTX and cubin slices from synthetic CUDA fatbin test fixtures. 3. Parse PTX assembly directives, outputting kernel entrypoints, parameters, and register budgets into structured JSON. 4. Extract cubin ELF sections and identify target SM architecture versions. 5. Correlate host `cudaLaunchKernel` invocations with target device kernel symbols. --- ## Related - #3620 [Feature] Shader and GPU Bytecode Reverse Engineering - #3624 [Feature] Native C++ semantic reconstruction - #3637 [Feature] ML model graph reverse engineering

bsakaguchi referenced this issue

2026-10-17 21:11:11 +00:00

[PR #1411] [CLOSED] fix(browser): resolve authorized scope when Chrome masks frame URLs #2865

bsakaguchi referenced this issue

2026-10-17 21:11:22 +00:00

[PR #1432] [CLOSED] fix(browser): launch verification Chrome without the desktop keyring #2881