[GH-ISSUE #3653] [Feature] LLM tokenizer reverse engineering: tokenizer.json, SentencePiece, BPE/Unigram vocabularies, special tokens and chat-template reconstruction #1374

New issue

Open

opened 2026-10-16 21:04:47 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:47 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3653

Summary

In modern Large Language Models (LLMs), the Tokenizer is the foundational translator between human language and neural network token embeddings. A model's behavior, multilingual performance, prompt length, token boundary vulnerabilities (token smuggling, Glitch tokens), and chat formatting are directly dictated by its tokenizer.

Production models distribute tokenizers in several formats:

Hugging Face tokenizer.json (Fast Tokenizers format);
Google SentencePiece binary models (tokenizer.model, based on Protocol Buffers);
Legacy Byte-Pair Encoding (BPE) vocabulary and merge files (vocab.json + merges.txt);
Tiktoken binary rank files (.tiktoken).

While issue #3323 (inspect_model_artifact) extracts basic tokenizer type and embedded chat templates from GGUF metadata, it does not inspect full external tokenizer files or analyze vocabulary topologies.

This feature proposal introduces dedicated LLM Tokenizer Reverse Engineering to REA, enabling static inspection of tokenizer.json, SentencePiece .model, and Tiktoken files, reconstructing token vocabularies, special tokens, pre-tokenization regexes, and Jinja chat templates.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Tokenizer File (tokenizer.json,  |
                                    |        tokenizer.model, vocab.json)     |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Hugging Face Fast         |                                   | Google SentencePiece      |
           | Tokenizer (tokenizer.json)|                                   | Model (Protobuf .model)   |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Vocabulary & Merge Rule Inversion     |
                                     | (BPE, WordPiece, Unigram, Byte-Level) |
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Special & Added Tokens    |                                   | Chat Template & Prompt    |
           | (BOS, EOS, PAD, FIM, Mask)|                                   | Formatting (Jinja2 AST)   |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Tokenizer Dossier,      |
                                     | Vocabulary Metrics & Prompt Structure |
                                     +---------------------------------------+

1. Tokenizer Format Ingestion

Hugging Face tokenizer.json: Parse the unified Fast Tokenizer JSON schema: normalizer, pre_tokenizer, model, post_processor, decoder, and added_tokens.
Google SentencePiece (tokenizer.model): Parse the Protocol Buffers ModelProto specification, extracting trainer specifications, normalizer specs, and piece pieces/scores.
OpenAI Tiktoken (.tiktoken): Parse base64-encoded token and rank files.
Legacy BPE (vocab.json + merges.txt): Parse vocabulary dictionaries and ordered merge rule lists.

2. Vocabulary & Model Algorithm Analysis

Model Type Identification: Classify the underlying tokenization algorithm:
- Byte-Pair Encoding (BPE)
- WordPiece
- Unigram (SentencePiece language model)
- Character / Byte-Level fallback.
Vocabulary Metrics:
- Total vocabulary size.
- Character coverage ratio.
- Longest tokens, token length distributions, and subword compression efficiency metrics.
Pre-tokenization Regular Expressions: Extract the splitting regex patterns (e.g. GPT-4 / LLaMA 3 regex patterns separating contractions, letters, numbers, and whitespace).

3. Special Tokens & Marker Inventory

Reserved Token Catalog: Extract and categorize special tokens and their exact integer token IDs:
- Beginning of Sequence (BOS), End of Sequence (EOS), Padding (PAD), Unknown (UNK).
- Fill-in-the-Middle (FIM) tokens: <|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>.
- Tool / Function Calling tokens: e.g. <|start_header_id|>, <|end_header_id|>, <tools>, </tools>.
- Vision / Multimodal markers: <image>, <video>, <|quad_start|>.

4. Chat Template & Formatter Inversion

Jinja2 Chat Template Parsing: Statically parse the embedded chat_template string in tokenizer_config.json or model metadata:
- Role mappings: system, user, assistant, tool.
- Header delimiters and turn delimiters.
- Generation prompt suffixes (e.g. `<|start_header_id|>assistant<|end_header_id|>

`).

Template Simulation: Test formatting of a standard dialogue conversation (system, user, assistant) against the recovered template to produce the exact prompt byte stream.

Acceptance Criteria

Introduce analyze_tokenizer in REA CLI and MCP contracts, accepting tokenizer.json, tokenizer.model, and vocab.json files.
Ingest synthetic Hugging Face tokenizer.json and SentencePiece tokenizer.model fixtures without error.
Extract vocabulary size, algorithm type, and merge counts into structured JSON evidence.
Identify all added and special tokens with their exact token IDs.
Invert and simulate the Jinja2 chat template, outputting formatted prompt boundaries.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3637 [Feature] ML model graph reverse engineering
#3608 [RFC/Feature] React Semantic Reconstruction & Decompiler Engine

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3653 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary In modern Large Language Models (LLMs), the **Tokenizer** is the foundational translator between human language and neural network token embeddings. A model's behavior, multilingual performance, prompt length, token boundary vulnerabilities (token smuggling, Glitch tokens), and chat formatting are directly dictated by its tokenizer. Production models distribute tokenizers in several formats: - Hugging Face `tokenizer.json` (Fast Tokenizers format); - Google SentencePiece binary models (`tokenizer.model`, based on Protocol Buffers); - Legacy Byte-Pair Encoding (BPE) vocabulary and merge files (`vocab.json` + `merges.txt`); - Tiktoken binary rank files (`.tiktoken`). While issue #3323 (`inspect_model_artifact`) extracts basic tokenizer type and embedded chat templates from GGUF metadata, it does not inspect full external tokenizer files or analyze vocabulary topologies. This feature proposal introduces dedicated **LLM Tokenizer Reverse Engineering** to REA, enabling static inspection of `tokenizer.json`, SentencePiece `.model`, and Tiktoken files, reconstructing token vocabularies, special tokens, pre-tokenization regexes, and Jinja chat templates. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Tokenizer File (tokenizer.json, | | tokenizer.model, vocab.json) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Hugging Face Fast | | Google SentencePiece | | Tokenizer (tokenizer.json)| | Model (Protobuf .model) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Vocabulary & Merge Rule Inversion | | (BPE, WordPiece, Unigram, Byte-Level) | +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Special & Added Tokens | | Chat Template & Prompt | | (BOS, EOS, PAD, FIM, Mask)| | Formatting (Jinja2 AST) | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Tokenizer Dossier, | | Vocabulary Metrics & Prompt Structure | +---------------------------------------+ ``` ### 1. Tokenizer Format Ingestion - **Hugging Face `tokenizer.json`:** Parse the unified Fast Tokenizer JSON schema: `normalizer`, `pre_tokenizer`, `model`, `post_processor`, `decoder`, and `added_tokens`. - **Google SentencePiece (`tokenizer.model`):** Parse the Protocol Buffers `ModelProto` specification, extracting trainer specifications, normalizer specs, and piece pieces/scores. - **OpenAI Tiktoken (`.tiktoken`):** Parse base64-encoded token and rank files. - **Legacy BPE (`vocab.json` + `merges.txt`):** Parse vocabulary dictionaries and ordered merge rule lists. ### 2. Vocabulary & Model Algorithm Analysis - **Model Type Identification:** Classify the underlying tokenization algorithm: - Byte-Pair Encoding (BPE) - WordPiece - Unigram (SentencePiece language model) - Character / Byte-Level fallback. - **Vocabulary Metrics:** - Total vocabulary size. - Character coverage ratio. - Longest tokens, token length distributions, and subword compression efficiency metrics. - **Pre-tokenization Regular Expressions:** Extract the splitting regex patterns (e.g. GPT-4 / LLaMA 3 regex patterns separating contractions, letters, numbers, and whitespace). ### 3. Special Tokens & Marker Inventory - **Reserved Token Catalog:** Extract and categorize special tokens and their exact integer token IDs: - Beginning of Sequence (`BOS`), End of Sequence (`EOS`), Padding (`PAD`), Unknown (`UNK`). - Fill-in-the-Middle (`FIM`) tokens: `<|fim_prefix|>`, `<|fim_middle|>`, `<|fim_suffix|>`. - Tool / Function Calling tokens: e.g. `<|start_header_id|>`, `<|end_header_id|>`, `<tools>`, `</tools>`. - Vision / Multimodal markers: `<image>`, `<video>`, `<|quad_start|>`. ### 4. Chat Template & Formatter Inversion - **Jinja2 Chat Template Parsing:** Statically parse the embedded `chat_template` string in `tokenizer_config.json` or model metadata: - Role mappings: `system`, `user`, `assistant`, `tool`. - Header delimiters and turn delimiters. - Generation prompt suffixes (e.g. `<|start_header_id|>assistant<|end_header_id|> `). - **Template Simulation:** Test formatting of a standard dialogue conversation (`system`, `user`, `assistant`) against the recovered template to produce the exact prompt byte stream. --- ## Acceptance Criteria 1. Introduce `analyze_tokenizer` in REA CLI and MCP contracts, accepting `tokenizer.json`, `tokenizer.model`, and `vocab.json` files. 2. Ingest synthetic Hugging Face `tokenizer.json` and SentencePiece `tokenizer.model` fixtures without error. 3. Extract vocabulary size, algorithm type, and merge counts into structured JSON evidence. 4. Identify all added and special tokens with their exact token IDs. 5. Invert and simulate the Jinja2 chat template, outputting formatted prompt boundaries. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3637 [Feature] ML model graph reverse engineering - #3608 [RFC/Feature] React Semantic Reconstruction & Decompiler Engine

bsakaguchi referenced this issue

2026-10-17 21:10:50 +00:00

[PR #1374] [MERGED] fix(ghidra): verify dense jump tables on an optimized fixture #2836

bsakaguchi referenced this issue

2026-10-17 21:10:55 +00:00

[PR #1382] [MERGED] fix(ghidra): keep the cross-format branch callee multi-block #2842