[GH-ISSUE #3680] [Feature] LLM inference-trace reverse engineering: token streams, KV-cache behavior, speculative decoding and generation-phase reconstruction #1390

New issue

Open

opened 2026-10-16 21:05:02 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:02 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3680

Summary

In modern Large Language Model (LLM) serving systems (vLLM, TGI, SGLang, TensorRT-LLM, llama.cpp), generation performance and cost are governed by complex runtime dynamics:

Prefill vs. Decode Phases: Time-to-First-Token (TTFT) computation vs. memory-bound autoregressive token generation;
KV-Cache Lifecycle: PagedAttention allocations, cache reuse, prefix caching, and cache eviction;
Speculative Decoding: Small draft models proposing candidate tokens verified in parallel by the target model;
Continuous Batching & Chunked Prefill: Dynamic scheduling of requests across iterations.

When auditing deployed AI systems, diagnosing performance anomalies, or analyzing captured inference traces (OTLP traces, server logs, or packet captures of model serving endpoints), analysts need to inspect how the model actually generated tokens.

This feature proposal introduces dedicated LLM Inference-Trace Reverse Engineering to REA, enabling automated parsing of inference execution traces, reconstruction of the prefill/decode phases, analysis of KV-cache memory dynamics, and verification of speculative decoding efficiency.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Inference Trace / Server Log     |
                                    |        (JSONL, OpenTelemetry, PCAP)     |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Token Stream Timeline     |                                   | Generation Phase Inversion|
           | Reconstructor (IDs, ms)   |                                   | (Prefill vs Decode Steps) |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | KV-Cache Dynamics & Memory Profiler   |
                                     | (PagedAttention Blocks, Prefix Cache) |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Speculative Decoding      |                                   | Latency & Throughput      |
           | Efficiency Analyzer       |                                   | Metric Synthesis (TTFT)   |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Generation Dossier,     |
                                     | Timeline Diagram & Cache Efficiency   |
                                     +---------------------------------------+

1. Token Stream & Timeline Ingestion

Trace Ingestion: Parse inference stream logs (Server-Sent Events / SSE streams data: {choices: [{"delta": ...}]}, OpenTelemetry trace spans, and framework execution logs).
Token Timestamp Alignment: Reconstruct high-resolution timestamps for every emitted token, calculating inter-token latency (ITL).

2. Prefill vs. Decode Phase Inversion

Prefill Characterization: Measure prompt token count, prefill duration, Time-to-First-Token (TTFT), and compute throughput (tokens/second).
Decode Phase Tracking: Track token-by-token generation steps, identifying latency jitter, sampling bottlenecks, and generation completion reasons (stop, length, content_filter).

3. KV-Cache Lifecycle & Memory Dynamics

Prefix Caching Analysis: Detect cache hit rates when prompts share common system instructions or conversation prefixes.
PagedAttention Memory Allocation: Trace allocation of virtual KV cache memory blocks, identifying memory fragmentation and maximum concurrent context capacities.

4. Speculative Decoding Efficiency

Draft vs. Target Verification: For traces capturing speculative decoding (e.g. Medusa, speculative draft models):
- Measure draft acceptance rate (percentage of draft tokens accepted per verification pass).
- Calculate effective speedup ratio (k_{effective}) over autoregressive baseline.

Acceptance Criteria

Introduce analyze_inference_trace in REA CLI and MCP contracts, accepting JSONL traces and SSE stream dumps.
Reconstruct token-by-token generation timelines with accurate inter-token latency metrics.
Distinguish prefill and decode execution phases from trace timestamps.
Calculate prefix caching hit ratios and KV-cache allocation footprints.
Export interactive generation timelines and throughput metrics in structured JSON evidence.

#3668 [Feature] Local LLM runtime reverse engineering
#3651 [Feature] LLM application reverse engineering
#3323 [Feature] Inspect GGUF and safetensors model files without loading them

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3680 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary In modern Large Language Model (LLM) serving systems (vLLM, TGI, SGLang, TensorRT-LLM, llama.cpp), generation performance and cost are governed by complex runtime dynamics: - **Prefill vs. Decode Phases:** Time-to-First-Token (TTFT) computation vs. memory-bound autoregressive token generation; - **KV-Cache Lifecycle:** PagedAttention allocations, cache reuse, prefix caching, and cache eviction; - **Speculative Decoding:** Small draft models proposing candidate tokens verified in parallel by the target model; - **Continuous Batching & Chunked Prefill:** Dynamic scheduling of requests across iterations. When auditing deployed AI systems, diagnosing performance anomalies, or analyzing captured inference traces (OTLP traces, server logs, or packet captures of model serving endpoints), analysts need to inspect how the model actually generated tokens. This feature proposal introduces dedicated **LLM Inference-Trace Reverse Engineering** to REA, enabling automated parsing of inference execution traces, reconstruction of the prefill/decode phases, analysis of KV-cache memory dynamics, and verification of speculative decoding efficiency. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Inference Trace / Server Log | | (JSONL, OpenTelemetry, PCAP) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Token Stream Timeline | | Generation Phase Inversion| | Reconstructor (IDs, ms) | | (Prefill vs Decode Steps) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | KV-Cache Dynamics & Memory Profiler | | (PagedAttention Blocks, Prefix Cache) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Speculative Decoding | | Latency & Throughput | | Efficiency Analyzer | | Metric Synthesis (TTFT) | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Generation Dossier, | | Timeline Diagram & Cache Efficiency | +---------------------------------------+ ``` ### 1. Token Stream & Timeline Ingestion - **Trace Ingestion:** Parse inference stream logs (Server-Sent Events / SSE streams `data: {choices: [{"delta": ...}]}`, OpenTelemetry trace spans, and framework execution logs). - **Token Timestamp Alignment:** Reconstruct high-resolution timestamps for every emitted token, calculating inter-token latency (ITL). ### 2. Prefill vs. Decode Phase Inversion - **Prefill Characterization:** Measure prompt token count, prefill duration, Time-to-First-Token (TTFT), and compute throughput (tokens/second). - **Decode Phase Tracking:** Track token-by-token generation steps, identifying latency jitter, sampling bottlenecks, and generation completion reasons (`stop`, `length`, `content_filter`). ### 3. KV-Cache Lifecycle & Memory Dynamics - **Prefix Caching Analysis:** Detect cache hit rates when prompts share common system instructions or conversation prefixes. - **PagedAttention Memory Allocation:** Trace allocation of virtual KV cache memory blocks, identifying memory fragmentation and maximum concurrent context capacities. ### 4. Speculative Decoding Efficiency - **Draft vs. Target Verification:** For traces capturing speculative decoding (e.g. Medusa, speculative draft models): - Measure draft acceptance rate (percentage of draft tokens accepted per verification pass). - Calculate effective speedup ratio ($k_{effective}$) over autoregressive baseline. --- ## Acceptance Criteria 1. Introduce `analyze_inference_trace` in REA CLI and MCP contracts, accepting JSONL traces and SSE stream dumps. 2. Reconstruct token-by-token generation timelines with accurate inter-token latency metrics. 3. Distinguish prefill and decode execution phases from trace timestamps. 4. Calculate prefix caching hit ratios and KV-cache allocation footprints. 5. Export interactive generation timelines and throughput metrics in structured JSON evidence. --- ## Related - #3668 [Feature] Local LLM runtime reverse engineering - #3651 [Feature] LLM application reverse engineering - #3323 [Feature] Inspect GGUF and safetensors model files without loading them

bsakaguchi referenced this issue

2026-10-17 21:01:32 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:10:58 +00:00

[PR #1390] [MERGED] fix(javascript): bound member-mutation paths through reassignment loops #2847

bsakaguchi referenced this issue

2026-10-17 21:11:12 +00:00

[PR #1414] [CLOSED] test(javascript): preserve member-alias fanout and loop regressions #2868