[GH-ISSUE #3651] [Feature] LLM application reverse engineering: system prompts, model/provider routing, tool schemas, RAG links and agent workflow reconstruction #1371

New issue

Open

opened 2026-10-16 21:04:46 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:46 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3651

Summary

Desktop clients, developer tools, CLI utilities, and web backends increasingly embed Large Language Model (LLM) agents and AI workflows. In AI application reverse engineering and security audits, analysts need to understand the underlying AI decision logic:

How the system prompt is assembled;
Which model provider endpoints and fallback chains are targeted;
Which external tools, function schemas, and MCP servers are declared;
How user inputs are augmented via Retrieval-Augmented Generation (RAG);
How the agent loops, handoffs, and tool results are orchestrated.

Currently, reverse engineering such applications requires tedious manual de-obfuscation across minified JavaScript bundles, Python wheels, or native executables.

This feature proposal introduces dedicated LLM Application Reverse Engineering to REA, enabling automated static extraction of system prompts, model routing pipelines, tool/function schemas, RAG vector retrievers, and end-to-end agent workflow graphs.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Application Artifact (ASAR, JS,  |
                                    |        Python wheel, native executable) |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | System Prompt Template    |                                   | Model & Provider Routing  |
           | Extractor (Jinja, strings)|                                   | Engine (OpenAI, Anthropic)|
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Tool & Function Schema Recovery       |
                                     | (JSON Schema, parameters, MCP servers)|
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | RAG Pipeline Inversion    |                                   | Agent Orchestration Graph |
           | (Embeddings, top-k, chunk)|                                   | (UI -> Prompt -> Tool loop|
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: AI Agent Architecture,  |
                                     | Prompts, Tool Specs & Flow Diagram    |
                                     +---------------------------------------+

1. System Prompt & Context Template Extraction

Template Discovery: Statically locate system prompts, developer prompts, and persona instructions embedded across template literals, string pools, or companion files (.prompt, .jinja, .yaml).
Dynamic Slot Analysis: Identify template interpolation placeholders (e.g. {{user_name}}, {{current_date}}, {{retrieved_context}}, {{tools}}), reconstructing the assembled prompt layout.
Safety & Guardrail Inversion: Extract embedded guardrails, refusal rules, content policy strings, and output formatting constraints (e.g. "Output strictly in JSON").

2. Model & Provider Routing Architecture

Provider Resolution: Identify target LLM provider APIs:
- OpenAI / Azure OpenAI endpoints (/v1/chat/completions).
- Anthropic Messages API (/v1/messages).
- Google Gemini API (generateContent).
- Local endpoints (Ollama, vLLM, llama.cpp, LocalAI).
Configuration Parameters: Extract default sampling parameters: model names (gpt-4o, claude-3-5-sonnet, gemini-1.5-pro), temperature, top_p, presence_penalty, frequency_penalty, and max_tokens.
Fallback Chains: Trace model failover logic (e.g. Primary Model -> Fallback Model on 429/500 errors).

3. Tool & Function Calling Schema Recovery

Function Definitions: Statically extract tool/function definitions declared in JSON Schema or Zod schemas:
- Function name, description, parameters, required fields, and parameter descriptions.
Tool Implementation Binding: Correlate declared tools with their actual local implementation handlers (e.g. shell execution, file read/write, database queries, web scraping).
MCP Server Configuration: Extract declarations of Model Context Protocol (MCP) server connections, command invocations, and capability handshakes.

4. RAG (Retrieval-Augmented Generation) Inversion

Vector Pipeline: Identify embedding model endpoints (e.g. text-embedding-3-small, HuggingFace models).
Retrieval Parameters: Extract similarity metric configurations (Cosine, Dot Product, L2), top-k values, score threshold cutoffs, and metadata filters.
Chunking Strategies: Recover chunk size and overlap parameters used during document ingestion.

5. Agent Orchestration Graph

Workflow State Machine: Build a directed workflow graph mapping:
User Input -> Prompt Construction -> LLM Inference -> Tool Call Dispatch -> Tool Result Injection -> Final Response.
Multi-Agent Handoffs: Detect multi-agent orchestration architectures (e.g. Planner -> Coder -> Reviewer) and inter-agent communication channels.

Acceptance Criteria

Introduce analyze_llm_application in REA CLI and MCP contracts.
Accurately extract system prompt templates and dynamic variable slots from synthetic application fixtures.
Recover declared tool calling JSON schemas and identify their backing JavaScript/Python function handlers.
Extract model provider endpoints, default model IDs, and sampling hyperparameters.
Export an end-to-end agent workflow diagram in structured JSON evidence.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3608 [RFC/Feature] React Semantic Reconstruction & Decompiler Engine
#3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3651 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Desktop clients, developer tools, CLI utilities, and web backends increasingly embed Large Language Model (LLM) agents and AI workflows. In AI application reverse engineering and security audits, analysts need to understand the underlying AI decision logic: - How the system prompt is assembled; - Which model provider endpoints and fallback chains are targeted; - Which external tools, function schemas, and MCP servers are declared; - How user inputs are augmented via Retrieval-Augmented Generation (RAG); - How the agent loops, handoffs, and tool results are orchestrated. Currently, reverse engineering such applications requires tedious manual de-obfuscation across minified JavaScript bundles, Python wheels, or native executables. This feature proposal introduces dedicated **LLM Application Reverse Engineering** to REA, enabling automated static extraction of system prompts, model routing pipelines, tool/function schemas, RAG vector retrievers, and end-to-end agent workflow graphs. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Application Artifact (ASAR, JS, | | Python wheel, native executable) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | System Prompt Template | | Model & Provider Routing | | Extractor (Jinja, strings)| | Engine (OpenAI, Anthropic)| +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Tool & Function Schema Recovery | | (JSON Schema, parameters, MCP servers)| +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | RAG Pipeline Inversion | | Agent Orchestration Graph | | (Embeddings, top-k, chunk)| | (UI -> Prompt -> Tool loop| +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: AI Agent Architecture, | | Prompts, Tool Specs & Flow Diagram | +---------------------------------------+ ``` ### 1. System Prompt & Context Template Extraction - **Template Discovery:** Statically locate system prompts, developer prompts, and persona instructions embedded across template literals, string pools, or companion files (`.prompt`, `.jinja`, `.yaml`). - **Dynamic Slot Analysis:** Identify template interpolation placeholders (e.g. `{{user_name}}`, `{{current_date}}`, `{{retrieved_context}}`, `{{tools}}`), reconstructing the assembled prompt layout. - **Safety & Guardrail Inversion:** Extract embedded guardrails, refusal rules, content policy strings, and output formatting constraints (e.g. "Output strictly in JSON"). ### 2. Model & Provider Routing Architecture - **Provider Resolution:** Identify target LLM provider APIs: - OpenAI / Azure OpenAI endpoints (`/v1/chat/completions`). - Anthropic Messages API (`/v1/messages`). - Google Gemini API (`generateContent`). - Local endpoints (Ollama, vLLM, llama.cpp, LocalAI). - **Configuration Parameters:** Extract default sampling parameters: model names (`gpt-4o`, `claude-3-5-sonnet`, `gemini-1.5-pro`), temperature, top_p, presence_penalty, frequency_penalty, and max_tokens. - **Fallback Chains:** Trace model failover logic (e.g. Primary Model -> Fallback Model on 429/500 errors). ### 3. Tool & Function Calling Schema Recovery - **Function Definitions:** Statically extract tool/function definitions declared in JSON Schema or Zod schemas: - Function name, description, parameters, required fields, and parameter descriptions. - **Tool Implementation Binding:** Correlate declared tools with their actual local implementation handlers (e.g. shell execution, file read/write, database queries, web scraping). - **MCP Server Configuration:** Extract declarations of Model Context Protocol (MCP) server connections, command invocations, and capability handshakes. ### 4. RAG (Retrieval-Augmented Generation) Inversion - **Vector Pipeline:** Identify embedding model endpoints (e.g. `text-embedding-3-small`, HuggingFace models). - **Retrieval Parameters:** Extract similarity metric configurations (Cosine, Dot Product, L2), top-$k$ values, score threshold cutoffs, and metadata filters. - **Chunking Strategies:** Recover chunk size and overlap parameters used during document ingestion. ### 5. Agent Orchestration Graph - **Workflow State Machine:** Build a directed workflow graph mapping: `User Input -> Prompt Construction -> LLM Inference -> Tool Call Dispatch -> Tool Result Injection -> Final Response`. - **Multi-Agent Handoffs:** Detect multi-agent orchestration architectures (e.g. Planner -> Coder -> Reviewer) and inter-agent communication channels. --- ## Acceptance Criteria 1. Introduce `analyze_llm_application` in REA CLI and MCP contracts. 2. Accurately extract system prompt templates and dynamic variable slots from synthetic application fixtures. 3. Recover declared tool calling JSON schemas and identify their backing JavaScript/Python function handlers. 4. Extract model provider endpoints, default model IDs, and sampling hyperparameters. 5. Export an end-to-end agent workflow diagram in structured JSON evidence. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3608 [RFC/Feature] React Semantic Reconstruction & Decompiler Engine - #3599 [Feature] Passive DNS extraction and cybersecurity analysis primitives

bsakaguchi referenced this issue

2026-10-17 21:01:31 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:05:46 +00:00

[GH-ISSUE #4806] [Bug] Default TOON output crashes with RangeError on a large resource_constraint that --format json returns #2498

bsakaguchi referenced this issue

2026-10-31 00:44:42 +00:00

[GH-ISSUE #6447] [Bug] CLI --token-count measures a different document from emitted JSON #3860