morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 07:47:29 +00:00

[GH-ISSUE #3668] [Feature] Local LLM runtime reverse engineering: llama.cpp, Ollama, MLX, ONNX Runtime, ExecuTorch and OpenVINO inference reconstruction #1379

New issue

Open

opened 2026-10-16 21:04:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:53 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3668

Summary

Local LLM and on-device AI runtimes (such as llama.cpp, Ollama, Apple MLX, ONNX Runtime, PyTorch ExecuTorch, and Intel OpenVINO) power an explosion of private, offline AI desktop applications, code assistants, and mobile assistants. In binary reverse engineering and vulnerability research, analysts encounter compiled applications embedding these runtimes into native executables, shared libraries (llama.dll, libonnxruntime.so, libmlx.dylib), or Electron sidecars.

While issue #3323 inspects passive GGUF model files on disk, it does not analyze the active native inference engine executing the model.

This feature proposal introduces dedicated Local LLM Runtime Reverse Engineering to REA, enabling automated static extraction of runtime engine identities, model loading configurations, GPU offload layers, context memory allocations, sampling pipelines, and native C/C++ inference API bindings.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Native Executable / Shared Lib   |
                                    |        (llama.dll, onnxruntime.so, app) |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Runtime Engine Signature  |                                   | Model Loading & Backend   |
           | Identifier (llama, MLX)   |                                   | Resolution (CUDA, Metal)  |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Context Window & Memory Allocator     |
                                     | (KV Cache, n_ctx, n_batch, VRAM)      |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Sampling Pipeline Parser  |                                   | Native API & Sidecar      |
           | (Temperature, Mirostat)   |                                   | Interface Mapping (IPC)   |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Runtime Configuration,  |
                                     | Compute Targets & Model Bindings      |
                                     +---------------------------------------+

1. Runtime Engine Signature Identification

Engine Classification: Statically identify embedded inference engines in PE/ELF/Mach-O binaries:
- llama.cpp / ggml: string signatures, exports (llama_backend_init, llama_model_load_from_file, llama_new_context_with_model).
- Ollama: Go sidecar binaries, API routes (/api/generate, /api/chat), Modelfile parser bindings.
- Apple MLX: Mach-O frameworks, Metal compute graph signatures.
- ONNX Runtime: OrtCreateEnv, OrtCreateSession, provider libraries (onnxruntime_providers_cuda.dll).
- PyTorch ExecuTorch: PTE runtime calls and operator registries.
- Intel OpenVINO: Core engine symbols (ov::Core, read_model, compile_model).

2. Model Loading & Hardware Backend Resolution

Target Backend: Detect enabled hardware acceleration backends:
- CUDA / cuBLAS / FlashAttention.
- Apple Metal (MPS / Metal Performance Shaders).
- Vulkan / OpenCL / Kompute.
- CPU acceleration: AVX-512, AVX2, ARM NEON.
GPU Offloading Parameters: Trace parameters passed to model loaders:
- n_gpu_layers: number of transformer layers offloaded to GPU VRAM vs. CPU RAM.
- main_gpu and tensor split ratios across multi-GPU setups.

3. Context Window & KV Cache Memory Profiling

Context Allocation Parameters: Recover default context sizes:
- n_ctx: context window token limit (e.g. 2048, 4096, 32768, 131072).
- n_batch: prompt processing batch size.
- n_ubatch: physical micro-batch size.
- kv_cache_type: FP16, Q8_0, or Q4_0 KV cache quantization.
Calculated Memory Footprint: Compute total runtime RAM/VRAM allocations required to initialize the model and KV cache.

4. Sampling Pipeline & Decoding Hyperparameters

Sampler Chain Inversion: Statically trace sampling configurations:
- Temperature, Top-k, Top-p (Nucleus sampling), Min-p.
- Repetition penalty, frequency and presence penalties.
- Mirostat sampling parameters (Mirostat v1/v2, \tau, \eta).
- Grammar constraints: embedded GBNF grammars or JSON Schema validators enforcing structured output.

5. Native API & IPC Sidecar Interfaces

Native C API Invocations: Invert function calls to the inference runtime, mapping eval, tokenize, detokenize, and token stream generation loops.
Sidecar IPC Endpoints: Map inter-process communication channels between Electron/UI wrappers and local runtime daemons (HTTP REST, Unix domain sockets, named pipes, gRPC).

Acceptance Criteria

Introduce analyze_local_llm_runtime in REA CLI and MCP contracts, accepting native binaries or application packages.
Accurately identify embedded llama.cpp, ONNX Runtime, and ExecuTorch engines from synthetic binary fixtures.
Extract configured context window sizes (n_ctx) and GPU layer offload counts.
Recover sampling hyperparameters (temperature, top_p, repetition penalties) and GBNF grammars.
Map IPC endpoints connecting frontends to local inference services.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3651 [Feature] LLM application reverse engineering
#3620 [Feature] Shader and GPU Bytecode Reverse Engineering

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3668 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Local LLM and on-device AI runtimes (such as **llama.cpp**, **Ollama**, Apple **MLX**, **ONNX Runtime**, PyTorch **ExecuTorch**, and Intel **OpenVINO**) power an explosion of private, offline AI desktop applications, code assistants, and mobile assistants. In binary reverse engineering and vulnerability research, analysts encounter compiled applications embedding these runtimes into native executables, shared libraries (`llama.dll`, `libonnxruntime.so`, `libmlx.dylib`), or Electron sidecars. While issue #3323 inspects passive GGUF model files on disk, it does not analyze the active native inference engine executing the model. This feature proposal introduces dedicated **Local LLM Runtime Reverse Engineering** to REA, enabling automated static extraction of runtime engine identities, model loading configurations, GPU offload layers, context memory allocations, sampling pipelines, and native C/C++ inference API bindings. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Native Executable / Shared Lib | | (llama.dll, onnxruntime.so, app) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Runtime Engine Signature | | Model Loading & Backend | | Identifier (llama, MLX) | | Resolution (CUDA, Metal) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Context Window & Memory Allocator | | (KV Cache, n_ctx, n_batch, VRAM) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Sampling Pipeline Parser | | Native API & Sidecar | | (Temperature, Mirostat) | | Interface Mapping (IPC) | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Runtime Configuration, | | Compute Targets & Model Bindings | +---------------------------------------+ ``` ### 1. Runtime Engine Signature Identification - **Engine Classification:** Statically identify embedded inference engines in PE/ELF/Mach-O binaries: - `llama.cpp` / `ggml`: string signatures, exports (`llama_backend_init`, `llama_model_load_from_file`, `llama_new_context_with_model`). - `Ollama`: Go sidecar binaries, API routes (`/api/generate`, `/api/chat`), Modelfile parser bindings. - Apple `MLX`: Mach-O frameworks, Metal compute graph signatures. - `ONNX Runtime`: `OrtCreateEnv`, `OrtCreateSession`, provider libraries (`onnxruntime_providers_cuda.dll`). - PyTorch `ExecuTorch`: PTE runtime calls and operator registries. - Intel `OpenVINO`: Core engine symbols (`ov::Core`, `read_model`, `compile_model`). ### 2. Model Loading & Hardware Backend Resolution - **Target Backend:** Detect enabled hardware acceleration backends: - CUDA / cuBLAS / FlashAttention. - Apple Metal (MPS / Metal Performance Shaders). - Vulkan / OpenCL / Kompute. - CPU acceleration: AVX-512, AVX2, ARM NEON. - **GPU Offloading Parameters:** Trace parameters passed to model loaders: - `n_gpu_layers`: number of transformer layers offloaded to GPU VRAM vs. CPU RAM. - `main_gpu` and tensor split ratios across multi-GPU setups. ### 3. Context Window & KV Cache Memory Profiling - **Context Allocation Parameters:** Recover default context sizes: - `n_ctx`: context window token limit (e.g. 2048, 4096, 32768, 131072). - `n_batch`: prompt processing batch size. - `n_ubatch`: physical micro-batch size. - `kv_cache_type`: FP16, Q8_0, or Q4_0 KV cache quantization. - **Calculated Memory Footprint:** Compute total runtime RAM/VRAM allocations required to initialize the model and KV cache. ### 4. Sampling Pipeline & Decoding Hyperparameters - **Sampler Chain Inversion:** Statically trace sampling configurations: - Temperature, Top-$k$, Top-$p$ (Nucleus sampling), Min-$p$. - Repetition penalty, frequency and presence penalties. - Mirostat sampling parameters (Mirostat v1/v2, $\tau$, $\eta$). - Grammar constraints: embedded GBNF grammars or JSON Schema validators enforcing structured output. ### 5. Native API & IPC Sidecar Interfaces - **Native C API Invocations:** Invert function calls to the inference runtime, mapping `eval`, `tokenize`, `detokenize`, and token stream generation loops. - **Sidecar IPC Endpoints:** Map inter-process communication channels between Electron/UI wrappers and local runtime daemons (HTTP REST, Unix domain sockets, named pipes, gRPC). --- ## Acceptance Criteria 1. Introduce `analyze_local_llm_runtime` in REA CLI and MCP contracts, accepting native binaries or application packages. 2. Accurately identify embedded `llama.cpp`, `ONNX Runtime`, and `ExecuTorch` engines from synthetic binary fixtures. 3. Extract configured context window sizes (`n_ctx`) and GPU layer offload counts. 4. Recover sampling hyperparameters (temperature, top_p, repetition penalties) and GBNF grammars. 5. Map IPC endpoints connecting frontends to local inference services. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3651 [Feature] LLM application reverse engineering - #3620 [Feature] Shader and GPU Bytecode Reverse Engineering

bsakaguchi referenced this issue

2026-10-17 21:10:55 +00:00

[PR #1379] [CLOSED] fix(setup): reject nonregular client configuration before reading #2840

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#1379

No description provided.

Rows
Columns