[GH-ISSUE #3682] [Feature] AI model package reverse engineering: Hugging Face repositories, shard indexes, configs, generation settings and component dependency graph #1395

New issue

Open

opened 2026-10-16 21:05:03 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:05:03 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3682

Problem Statement & Context

Modern open-weights AI models rarely ship as isolated single-file binaries. Instead, production model distributions—specifically Hugging Face hub repositories and local multi-file directories—are structured composite packages containing heterogeneous configuration files, weight shard indexes, tokenizer vocabularies, generation policies, preprocessor recipes, and sub-component directories (e.g. multimodal VLMs or Stable Diffusion pipelines).

Currently, reverse engineering an unknown or repackaged AI model directory requires manually piecing together:

config.json: Model architecture, layer counts, attention heads, RoPE theta, activation functions, tie word embeddings flag.
model.safetensors.index.json / pytorch_model.bin.index.json: Shard index maps locating tens to hundreds of weight files (model-00001-of-00030.safetensors) and total parameter sizes.
generation_config.json: Default decoding settings (temperature, top_p, top_k, repetition_penalty, eos_token_id, bos_token_id, max_length).
Sub-folder models: Multimodal Vision-Language Models (VLM) containing vision_tower, multi_modal_projector, and language_model, or Diffusion packages containing text_encoder, unet, transformer, vae, and scheduler.
tokenizer_config.json, chat_template.json, and added tokens (#3653).
Quantization configurations: quantization_config (bitsandbytes, AWQ, GPTQ, HQQ).

A unified package-level reverse engineering engine is required to ingest an entire model directory, audit its file integrity and weight references, resolve shard topologies, detect missing or dangling weights, and reconstruct the end-to-end component dependency graph.

Proposed Architecture

flowchart TD
    A["Hugging Face Model Directory / Package"] --> B["AI Model Package Ingestion Engine"]
    B --> C["Config & Shard Resolver"]
    B --> D["Multimodal Sub-Component Mapper"]
    B --> E["Generation & Execution Policy Extractor"]
    
    C --> C1["Shard Index Validation (safetensors.index.json)"]
    C --> C2["Tensor Location Mapping & Shard Balancing"]
    C --> C3["Dangling / Missing Weight Verification"]
    
    D --> D1["Composite Models (Vision Tower + Projector + LLM)"]
    D --> D2["Pipeline Stages (VAE, UNet/DiT, CLIP/T5)"]
    
    E --> E1["Decoding Hyperparameters & Stop Criteria"]
    E --> E2["Quantization Metadata (BitsAndBytes, AWQ, GPTQ)"]
    
    C & D & E --> F["Unified Model Bill of Materials (Model BOM) & Architectural Graph"]

Technical Capabilities & Modules

Repository Topology & Shard Index Verification:
- Parse model.safetensors.index.json and pytorch_model.bin.index.json.
- Validate 1:1 mapping between index entries and physical shard files on disk.
- Detect missing shard files, unindexed orphan weight files, and calculate exact per-shard parameter and byte distributions.
- Compute aggregate model parameter count partitioned by layer types (embeddings, attention, MLP, norm, heads).
Multimodal & Composite Component Graph Reconstruction:
- Detect nested sub-component architectures:
  - Vision-Language Models (LLaVA, Qwen-VL, PaliGemma, InternVL): map vision backbone, spatial pooling/downsampler, projector MLP, and causal language model.
  - Diffusion Pipelines (SD 1.5, SDXL, Flux, SD3): map text encoders (CLIP-L, CLIP-G, T5-XXL), latent autoencoder (VAE), denoising backbone (DiT/UNet), and scheduler.
- Reconstruct cross-component tensor links and dimensional interfaces (e.g. projector input dim = vision output dim, projector output dim = LLM hidden dim).
Unified Model Bill of Materials (Model BOM):
- Synthesize a comprehensive JSON/Markdown manifest detailing:
  - Exact base architecture family (LlamaForCausalLM, Qwen2VLForConditionalGeneration, FluxForEngineExecution).
  - Precision breakdown (% bf16, % fp16, % int4, % int8, % fp8).
  - Quantization schema and runtime backend compatibility.
  - Context window limits (max_position_embeddings, RoPE frequency scaling, YaRN configurations).
Generation & Decoding Rule Extraction:
- Reconcile generation_config.json with base config.json.
- Extract default sampling policies: temperature, top-k, top-p, min-p, typical-p, presence/frequency penalties.
- Extract multi-token stop sequences, EOS/BOS token ID mappings, and maximum generation horizons.
Security & Package Integrity Audit:
- Scan for legacy pickled PyTorch weights (pytorch_model.bin) using static AST analysis (#3652) to flag deserialization risks.
- Validate cryptographic checksums (sha256 in git lfs metadata or safetensors headers).

Acceptance Criteria

Complete inspection and dependency graph reconstruction for single-model and multimodal composite packages.
Shard index reconciliation verifying 100% of tensor references across physical .safetensors files without loading weights into RAM.
End-to-end Model BOM generation with layer-by-layer parameter and precision accounting.
Seamless integration with tokenizer (#3653), prompt template (#3679), and inference runtime (#3668) modules.

Complements #3323 (AI model reverse engineering: GGUF, Safetensors, architecture)
Complements #3652 (PyTorch checkpoint reverse engineering)
Complements #3653 (LLM tokenizer reverse engineering)
Complements #3669 (Model lineage reverse engineering: base-model fingerprinting, shard topology)

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3682 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: AI & Runtimes](https://img.shields.io/badge/domain-AI_&_runtimes-blueviolet.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ### Problem Statement & Context Modern open-weights AI models rarely ship as isolated single-file binaries. Instead, production model distributions—specifically Hugging Face hub repositories and local multi-file directories—are structured composite packages containing heterogeneous configuration files, weight shard indexes, tokenizer vocabularies, generation policies, preprocessor recipes, and sub-component directories (e.g. multimodal VLMs or Stable Diffusion pipelines). Currently, reverse engineering an unknown or repackaged AI model directory requires manually piecing together: - `config.json`: Model architecture, layer counts, attention heads, RoPE theta, activation functions, tie word embeddings flag. - `model.safetensors.index.json` / `pytorch_model.bin.index.json`: Shard index maps locating tens to hundreds of weight files (`model-00001-of-00030.safetensors`) and total parameter sizes. - `generation_config.json`: Default decoding settings (temperature, top_p, top_k, repetition_penalty, eos_token_id, bos_token_id, max_length). - Sub-folder models: Multimodal Vision-Language Models (VLM) containing `vision_tower`, `multi_modal_projector`, and `language_model`, or Diffusion packages containing `text_encoder`, `unet`, `transformer`, `vae`, and `scheduler`. - `tokenizer_config.json`, `chat_template.json`, and added tokens (#3653). - Quantization configurations: `quantization_config` (bitsandbytes, AWQ, GPTQ, HQQ). A unified package-level reverse engineering engine is required to ingest an entire model directory, audit its file integrity and weight references, resolve shard topologies, detect missing or dangling weights, and reconstruct the end-to-end component dependency graph. ### Proposed Architecture ```mermaid flowchart TD A["Hugging Face Model Directory / Package"] --> B["AI Model Package Ingestion Engine"] B --> C["Config & Shard Resolver"] B --> D["Multimodal Sub-Component Mapper"] B --> E["Generation & Execution Policy Extractor"] C --> C1["Shard Index Validation (safetensors.index.json)"] C --> C2["Tensor Location Mapping & Shard Balancing"] C --> C3["Dangling / Missing Weight Verification"] D --> D1["Composite Models (Vision Tower + Projector + LLM)"] D --> D2["Pipeline Stages (VAE, UNet/DiT, CLIP/T5)"] E --> E1["Decoding Hyperparameters & Stop Criteria"] E --> E2["Quantization Metadata (BitsAndBytes, AWQ, GPTQ)"] C & D & E --> F["Unified Model Bill of Materials (Model BOM) & Architectural Graph"] ``` ### Technical Capabilities & Modules 1. **Repository Topology & Shard Index Verification:** - Parse `model.safetensors.index.json` and `pytorch_model.bin.index.json`. - Validate 1:1 mapping between index entries and physical shard files on disk. - Detect missing shard files, unindexed orphan weight files, and calculate exact per-shard parameter and byte distributions. - Compute aggregate model parameter count partitioned by layer types (embeddings, attention, MLP, norm, heads). 2. **Multimodal & Composite Component Graph Reconstruction:** - Detect nested sub-component architectures: - Vision-Language Models (LLaVA, Qwen-VL, PaliGemma, InternVL): map vision backbone, spatial pooling/downsampler, projector MLP, and causal language model. - Diffusion Pipelines (SD 1.5, SDXL, Flux, SD3): map text encoders (CLIP-L, CLIP-G, T5-XXL), latent autoencoder (VAE), denoising backbone (DiT/UNet), and scheduler. - Reconstruct cross-component tensor links and dimensional interfaces (e.g. projector input dim = vision output dim, projector output dim = LLM hidden dim). 3. **Unified Model Bill of Materials (Model BOM):** - Synthesize a comprehensive JSON/Markdown manifest detailing: - Exact base architecture family (`LlamaForCausalLM`, `Qwen2VLForConditionalGeneration`, `FluxForEngineExecution`). - Precision breakdown (% bf16, % fp16, % int4, % int8, % fp8). - Quantization schema and runtime backend compatibility. - Context window limits (`max_position_embeddings`, RoPE frequency scaling, YaRN configurations). 4. **Generation & Decoding Rule Extraction:** - Reconcile `generation_config.json` with base `config.json`. - Extract default sampling policies: temperature, top-k, top-p, min-p, typical-p, presence/frequency penalties. - Extract multi-token stop sequences, EOS/BOS token ID mappings, and maximum generation horizons. 5. **Security & Package Integrity Audit:** - Scan for legacy pickled PyTorch weights (`pytorch_model.bin`) using static AST analysis (#3652) to flag deserialization risks. - Validate cryptographic checksums (`sha256` in git lfs metadata or safetensors headers). ### Acceptance Criteria - [ ] Complete inspection and dependency graph reconstruction for single-model and multimodal composite packages. - [ ] Shard index reconciliation verifying 100% of tensor references across physical `.safetensors` files without loading weights into RAM. - [ ] End-to-end Model BOM generation with layer-by-layer parameter and precision accounting. - [ ] Seamless integration with tokenizer (#3653), prompt template (#3679), and inference runtime (#3668) modules. ### Related Issues & Dependencies - Complements #3323 (AI model reverse engineering: GGUF, Safetensors, architecture) - Complements #3652 (PyTorch checkpoint reverse engineering) - Complements #3653 (LLM tokenizer reverse engineering) - Complements #3669 (Model lineage reverse engineering: base-model fingerprinting, shard topology)

bsakaguchi referenced this issue

2026-10-17 21:11:01 +00:00

[PR #1393] [MERGED] fix(javascript): stop reading media preload attributes as Electron preloads #2850

bsakaguchi referenced this issue

2026-10-17 21:11:02 +00:00

[PR #1395] [MERGED] fix(cli): stop advertising multi-letter single-dash option aliases #2852