[GH-ISSUE #3666] [Feature] Quantized LLM reverse engineering: GPTQ, AWQ, EXL2 and GGUF quantization reconstruction #1380

New issue

Open

opened 2026-10-16 21:04:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:53 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3666

Summary

To deploy massive Large Language Models (LLMs) on consumer GPUs, laptops, and mobile devices, models are compressed using post-training quantization techniques:

GGUF Quantization: k-quants (Q2_K, Q3_K, Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ2_XXS, IQ3_M, IQ4_NL);
GPTQ (Generalized Post-Training Quantization): 4-bit/8-bit integer quantization with Cholesky matrix second-order error compensation;
AWQ (Activation-aware Weight Quantization): Protecting salient weight channels based on activation magnitudes;
EXL2 (ExLlamaV2): Variable mixed-bitrate quantization (e.g. 2.2 to 8.0 bits per weight) per layer.

While issue #3323 inspects basic GGUF header tensor names and types, it does not deconstruct the packing layouts, group sizes, scale tensors, or calibration metadata of quantized weights.

This feature proposal introduces dedicated Quantized LLM Reverse Engineering to REA, enabling automated inspection of quantization schemes, group sizes, scale/zero-point packing formats, mixed precision layers, and runtime execution requirements across GPTQ, AWQ, EXL2, and GGUF.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Quantized Model Artifact         |
                                    |        (GGUF, safetensors, quant_config)|
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                               |                               |
                         v                               v                               v
           +---------------------------+   +---------------------------+   +---------------------------+
           | GGUF k-Quant / i-Quant    |   | GPTQ / AWQ Config &       |   | EXL2 Variable Bitrate     |
           | Format Inversion          |   | Packed Tensor Reader      |   | Schema Inversion          |
           +-------------+-------------+   +-------------+-------------+   +-------------+-------------+
                         |                               |                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Quantization Parameter Resolver       |
                                     | (Bits/Weight, Group Size, Scales, ZP) |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Packing Layout & Bitfield |                                   | Per-Layer Bitrate &       |
           | Unpacking (Interleaving)  |                                   | Memory Budget Calculator  |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Quantization Dossier,   |
                                     | Effective Bits, and VRAM Profiles     |
                                     +---------------------------------------+

1. Quantization Format Ingestion

GPTQ Format: Parse quant_config.json (bits, group_size, damp_percent, desc_act, sym), mapping qweight, qzeros, scales, and g_idx tensors.
AWQ Format: Parse AWQ metadata (bits, group_size, version), mapping qweight, qzeros, and scales.
EXL2 Format: Parse ExLlamaV2 config files and quantized safetensors arrays.
GGUF k-quants & i-quant: Deconstruct block structures for legacy (Q4_0, Q4_1, Q8_0), k-quants (Q4_K, Q5_K, Q6_K), and importance matrix quants (IQ1_S, IQ2_XXS, IQ3_M, IQ4_XS).

2. Packing Layout & Bitfield Inversion

Integer Packing: Unpack bit-interleaved tensor arrays:
- 32-bit packed integers containing eight 4-bit weights (int32 packing).
- Zero-point packing and bit-shifts (zero_point offsets).
- Activation order permutation arrays (g_idx).
GGUF Super-Block Layouts: Parse k-quant super-blocks: 256-value blocks split into sub-blocks with low-rank scale factors and quantized offsets.

3. Mixed Precision & Layer-by-Layer Bitrate Profiling

Per-Layer Profiling: Map effective bits-per-weight (bpw) across model layers:
- Distinguish critical layers kept at higher precision (e.g. embeddings embed_tokens, attention output v_proj/o_proj, and final lm_head at FP16/Q8_0) from aggressively quantized MLP layers (Q4_K/Q3_K).
Compression Efficiency Metric: Calculate actual compressed bits-per-parameter against the theoretical unquantized FP16 footprint.

4. Runtime Compatibility & Memory Profiling

Hardware Architecture Constraints: Identify required hardware acceleration capabilities (e.g. AVX-512 VNNI on CPU, Tensor Core INT4/INT8 on CUDA, Apple Silicon NEON).
VRAM Requirements: Calculate the exact memory footprint for model weights plus Context Length KV Cache allocations (2 \times 2 \times n_{layers} \times d_{head} \times n_{heads} \times ctx).

Acceptance Criteria

Introduce analyze_quantized_model in REA CLI and MCP contracts, accepting GGUF, GPTQ, and AWQ model directories or files.
Parse quant_config.json and document bits, group size, and symmetric/asymmetric quantization modes.
Decode GGUF k-quant tensor types (e.g. Q4_K_M) into their constituent sub-block quantization parameters.
Calculate layer-by-layer effective bits-per-weight and total model memory footprints.
Identify required compute kernel intrinsics and runtime compatibility requirements.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3637 [Feature] ML model graph reverse engineering
#3620 [Feature] Shader and GPU Bytecode Reverse Engineering

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3666 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary To deploy massive Large Language Models (LLMs) on consumer GPUs, laptops, and mobile devices, models are compressed using post-training quantization techniques: - **GGUF Quantization:** k-quants (Q2_K, Q3_K, Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ2_XXS, IQ3_M, IQ4_NL); - **GPTQ (Generalized Post-Training Quantization):** 4-bit/8-bit integer quantization with Cholesky matrix second-order error compensation; - **AWQ (Activation-aware Weight Quantization):** Protecting salient weight channels based on activation magnitudes; - **EXL2 (ExLlamaV2):** Variable mixed-bitrate quantization (e.g. 2.2 to 8.0 bits per weight) per layer. While issue #3323 inspects basic GGUF header tensor names and types, it does not deconstruct the packing layouts, group sizes, scale tensors, or calibration metadata of quantized weights. This feature proposal introduces dedicated **Quantized LLM Reverse Engineering** to REA, enabling automated inspection of quantization schemes, group sizes, scale/zero-point packing formats, mixed precision layers, and runtime execution requirements across **GPTQ**, **AWQ**, **EXL2**, and **GGUF**. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Quantized Model Artifact | | (GGUF, safetensors, quant_config)| +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | | v v v +---------------------------+ +---------------------------+ +---------------------------+ | GGUF k-Quant / i-Quant | | GPTQ / AWQ Config & | | EXL2 Variable Bitrate | | Format Inversion | | Packed Tensor Reader | | Schema Inversion | +-------------+-------------+ +-------------+-------------+ +-------------+-------------+ | | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Quantization Parameter Resolver | | (Bits/Weight, Group Size, Scales, ZP) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Packing Layout & Bitfield | | Per-Layer Bitrate & | | Unpacking (Interleaving) | | Memory Budget Calculator | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Quantization Dossier, | | Effective Bits, and VRAM Profiles | +---------------------------------------+ ``` ### 1. Quantization Format Ingestion - **GPTQ Format:** Parse `quant_config.json` (`bits`, `group_size`, `damp_percent`, `desc_act`, `sym`), mapping `qweight`, `qzeros`, `scales`, and `g_idx` tensors. - **AWQ Format:** Parse AWQ metadata (`bits`, `group_size`, `version`), mapping `qweight`, `qzeros`, and `scales`. - **EXL2 Format:** Parse ExLlamaV2 config files and quantized safetensors arrays. - **GGUF k-quants & i-quant:** Deconstruct block structures for legacy (Q4_0, Q4_1, Q8_0), k-quants (Q4_K, Q5_K, Q6_K), and importance matrix quants (IQ1_S, IQ2_XXS, IQ3_M, IQ4_XS). ### 2. Packing Layout & Bitfield Inversion - **Integer Packing:** Unpack bit-interleaved tensor arrays: - 32-bit packed integers containing eight 4-bit weights (`int32` packing). - Zero-point packing and bit-shifts (`zero_point` offsets). - Activation order permutation arrays (`g_idx`). - **GGUF Super-Block Layouts:** Parse k-quant super-blocks: 256-value blocks split into sub-blocks with low-rank scale factors and quantized offsets. ### 3. Mixed Precision & Layer-by-Layer Bitrate Profiling - **Per-Layer Profiling:** Map effective bits-per-weight (bpw) across model layers: - Distinguish critical layers kept at higher precision (e.g. embeddings `embed_tokens`, attention output `v_proj`/`o_proj`, and final `lm_head` at FP16/Q8_0) from aggressively quantized MLP layers (Q4_K/Q3_K). - **Compression Efficiency Metric:** Calculate actual compressed bits-per-parameter against the theoretical unquantized FP16 footprint. ### 4. Runtime Compatibility & Memory Profiling - **Hardware Architecture Constraints:** Identify required hardware acceleration capabilities (e.g. AVX-512 VNNI on CPU, Tensor Core INT4/INT8 on CUDA, Apple Silicon NEON). - **VRAM Requirements:** Calculate the exact memory footprint for model weights plus Context Length KV Cache allocations ($2 \times 2 \times n_{layers} \times d_{head} \times n_{heads} \times ctx$). --- ## Acceptance Criteria 1. Introduce `analyze_quantized_model` in REA CLI and MCP contracts, accepting GGUF, GPTQ, and AWQ model directories or files. 2. Parse `quant_config.json` and document bits, group size, and symmetric/asymmetric quantization modes. 3. Decode GGUF k-quant tensor types (e.g. Q4_K_M) into their constituent sub-block quantization parameters. 4. Calculate layer-by-layer effective bits-per-weight and total model memory footprints. 5. Identify required compute kernel intrinsics and runtime compatibility requirements. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3637 [Feature] ML model graph reverse engineering - #3620 [Feature] Shader and GPU Bytecode Reverse Engineering

bsakaguchi referenced this issue

2026-10-17 21:01:31 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:10:58 +00:00

[PR #1386] [MERGED] fix(javascript): handle oversized inline source maps #2845

bsakaguchi referenced this issue

2026-10-17 21:10:58 +00:00

[PR #1388] [CLOSED] fix(javascript): avoid regex overflow for large source maps #2846

bsakaguchi referenced this issue

2026-10-31 00:37:52 +00:00

[GH-ISSUE #5253] [Bug] Valid large base64 bodies overflow shared capture validation regexes #3208