CLIP wiring and scheduler reconstruction #1387

New issue

Open

opened 2026-10-16 21:04:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-16 21:04:59 +00:00

Owner

Originally created by @matbrgz on GitHub (Oct 14, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/3675

Summary

Latent diffusion models and flow-matching image/video generative models (such as Stable Diffusion 1.5/2.1, SDXL, Stable Video Diffusion, and modern FLUX.1 Diffusion Transformers) represent the dominant generative AI architecture for imagery and video. In production software, desktop AI studios (ComfyUI, Automatic1111), and mobile photo apps, these models are distributed as modular multi-network packages:

Diffusion Backbones: Convolutional UNet (UNet2DConditionModel) or Diffusion Transformer (DiT / MMDiT);
Text Encoders: OpenAI CLIP ViT-L/14, OpenCLIP ViT-bigG, and Google T5-v1.1-XXL;
Variational Autoencoders (VAE): AutoencoderKL compressing images into latent space;
Control & Adapter Add-ons: ControlNet, T2I-Adapter, IP-Adapter, LoRAs, and Textual Inversion embeddings;
Schedulers & Solvers: DDIM, Euler Ancestral, DPM++ 2M Karras, FlowMatchEulerDiscreteScheduler.

Currently, REA has no tooling to analyze generative diffusion pipelines or inspect ComfyUI workflow graphs.

This feature proposal introduces dedicated Diffusion Model Reverse Engineering to REA, enabling automated inspection of modular diffusion components, parameter alignment, scheduler configurations, and ComfyUI node graph reconstruction.

Technical Capabilities & Architecture

                                    +-----------------------------------------+
                                    | Input: Diffusion Model / Checkpoint     |
                                    |        (safetensors, ComfyUI workflow)  |
                                    +--------------------+--------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Checkpoint Component      |                                   | ComfyUI Visual Workflow   |
           | Classifier (UNet, DiT, VAE)                                   | Node Graph Parser (JSON)  |
           +-------------+-------------+                                   +-------------+-------------+
                         |                                                               |
                         +-------------------------------+-------------------------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | Architecture Parameter Inversion      |
                                     | (Attention Heads, Channels, CrossAttn)|
                                     +-------------------+-------------------+
                                                         |
                         +-------------------------------+-------------------------------+
                         |                                                               |
                         v                                                               v
           +---------------------------+                                   +---------------------------+
           | Conditioning & Text       |                                   | Scheduler & Sampling      |
           | Encoder Wiring (CLIP/T5)  |                                   | Configuration (Timesteps) |
           +-------------+-------------+                                   +-------------+-------------+
                                                         |
                                                         v
                                     +---------------------------------------+
                                     | REA Evidence: Pipeline Architecture,  |
                                     | Component Wiring & Generation Recipe  |
                                     +---------------------------------------+

1. Checkpoint Component De-Multiplexing

Tensor Prefix Analysis: Dissect monolithic checkpoints (.safetensors, .ckpt) by prefix namespace:
- model.diffusion_model.*: UNet or DiT backbone.
- cond_stage_model.* / text_encoders.*: CLIP or T5 text encoder.
- first_stage_model.* / vae.*: Latent VAE autoencoder.
Standalone Component Ingestion: Parse standard Hugging Face diffusers directory repositories (model_index.json, subdirectories unet/, transformer/, vae/, text_encoder/, scheduler/).

2. Diffusion Backbone Architecture Recognition

Architecture Classification: Automatically classify the diffusion backbone:
- UNet (SD 1.5, SD 2.1, SDXL): cross-attention dimensions, block channels, down/up-sampling levels.
- Diffusion Transformer (DiT / MMDiT in FLUX.1, SD3): joint text-image attention blocks, modulation layers, rotary position embeddings.
ControlNet & IP-Adapter Discovery: Detect zero-convolution residual weights (controlnet_down_blocks) and cross-attention image projection layers (ip_adapter).

3. Text Encoder & Conditioning Wiring

Context Dimension Matching: Extract the cross-attention projection dimension (e.g. 768 for CLIP-L, 1280 for OpenCLIP-G, 4096 for T5-XXL), determining the required text encoder pair.
Textual Inversion Embeddings: Detect embedded trigger token vectors and token replacement rules.

4. Scheduler & Numerical Solver Reconstruction

Scheduler Extraction: Parse scheduler/scheduler_config.json parameters:
- Number of training timesteps (e.g. 1000).
- Beta schedule (linear, scaled_linear), \beta_{start}, \beta_{end}.
- Prediction type (epsilon, v_prediction, flow_prediction).
- Sampling algorithm compatibility (Euler, Heun, DPM-Solver).

5. ComfyUI Workflow Graph Inversion

Embedded PNG/JSON Metadata: Extract embedded workflow and prompt JSON metadata stored inside generated PNG tEXt chunks or standalone .json workflows.
Execution DAG Inversion: Reconstruct the node graph connecting:
CheckpointLoader -> CLIPTextEncode (Prompt/Negative) -> KSampler -> VAEDecode -> SaveImage.

Acceptance Criteria

Introduce analyze_diffusion_pipeline in REA CLI and MCP contracts, accepting diffusion checkpoints and ComfyUI workflow JSONs.
Classify monolithic diffusion checkpoints into UNet, DiT, VAE, and Text Encoder components without loading weights into RAM.
Extract cross-attention conditioning dimensions and identify compatible text encoders.
Parse scheduler configs and report timesteps, beta schedules, and prediction targets.
Invert embedded ComfyUI workflows into structured JSON pipeline graphs.

#3323 [Feature] Inspect GGUF and safetensors model files without loading them
#3665 [Feature] LoRA/PEFT adapter reverse engineering
#3652 [Feature] PyTorch checkpoint reverse engineering

Originally created by @matbrgz on GitHub (Oct 14, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/3675 [![Type: Feature](https://img.shields.io/badge/type-feature-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Domain: Game & Engine RE](https://img.shields.io/badge/domain-game_&_engine_RE-green.svg?style=flat-square)](https://github.com/morluto/rea) [![Status: Proposed](https://img.shields.io/badge/status-proposed-blue.svg?style=flat-square)](https://github.com/morluto/rea) [![Ecosystem: REA](https://img.shields.io/badge/ecosystem-morluto%2Frea-8A2BE2.svg?style=flat-square)](https://github.com/morluto/rea) ## Summary Latent diffusion models and flow-matching image/video generative models (such as Stable Diffusion 1.5/2.1, SDXL, Stable Video Diffusion, and modern FLUX.1 Diffusion Transformers) represent the dominant generative AI architecture for imagery and video. In production software, desktop AI studios (ComfyUI, Automatic1111), and mobile photo apps, these models are distributed as modular multi-network packages: - **Diffusion Backbones:** Convolutional UNet (`UNet2DConditionModel`) or Diffusion Transformer (DiT / MMDiT); - **Text Encoders:** OpenAI CLIP ViT-L/14, OpenCLIP ViT-bigG, and Google T5-v1.1-XXL; - **Variational Autoencoders (VAE):** AutoencoderKL compressing images into latent space; - **Control & Adapter Add-ons:** ControlNet, T2I-Adapter, IP-Adapter, LoRAs, and Textual Inversion embeddings; - **Schedulers & Solvers:** DDIM, Euler Ancestral, DPM++ 2M Karras, FlowMatchEulerDiscreteScheduler. Currently, REA has no tooling to analyze generative diffusion pipelines or inspect ComfyUI workflow graphs. This feature proposal introduces dedicated **Diffusion Model Reverse Engineering** to REA, enabling automated inspection of modular diffusion components, parameter alignment, scheduler configurations, and ComfyUI node graph reconstruction. --- ## Technical Capabilities & Architecture ``` +-----------------------------------------+ | Input: Diffusion Model / Checkpoint | | (safetensors, ComfyUI workflow) | +--------------------+--------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Checkpoint Component | | ComfyUI Visual Workflow | | Classifier (UNet, DiT, VAE) | Node Graph Parser (JSON) | +-------------+-------------+ +-------------+-------------+ | | +-------------------------------+-------------------------------+ | v +---------------------------------------+ | Architecture Parameter Inversion | | (Attention Heads, Channels, CrossAttn)| +-------------------+-------------------+ | +-------------------------------+-------------------------------+ | | v v +---------------------------+ +---------------------------+ | Conditioning & Text | | Scheduler & Sampling | | Encoder Wiring (CLIP/T5) | | Configuration (Timesteps) | +-------------+-------------+ +-------------+-------------+ | v +---------------------------------------+ | REA Evidence: Pipeline Architecture, | | Component Wiring & Generation Recipe | +---------------------------------------+ ``` ### 1. Checkpoint Component De-Multiplexing - **Tensor Prefix Analysis:** Dissect monolithic checkpoints (`.safetensors`, `.ckpt`) by prefix namespace: - `model.diffusion_model.*`: UNet or DiT backbone. - `cond_stage_model.*` / `text_encoders.*`: CLIP or T5 text encoder. - `first_stage_model.*` / `vae.*`: Latent VAE autoencoder. - **Standalone Component Ingestion:** Parse standard Hugging Face `diffusers` directory repositories (`model_index.json`, subdirectories `unet/`, `transformer/`, `vae/`, `text_encoder/`, `scheduler/`). ### 2. Diffusion Backbone Architecture Recognition - **Architecture Classification:** Automatically classify the diffusion backbone: - UNet (SD 1.5, SD 2.1, SDXL): cross-attention dimensions, block channels, down/up-sampling levels. - Diffusion Transformer (DiT / MMDiT in FLUX.1, SD3): joint text-image attention blocks, modulation layers, rotary position embeddings. - **ControlNet & IP-Adapter Discovery:** Detect zero-convolution residual weights (`controlnet_down_blocks`) and cross-attention image projection layers (`ip_adapter`). ### 3. Text Encoder & Conditioning Wiring - **Context Dimension Matching:** Extract the cross-attention projection dimension (e.g. 768 for CLIP-L, 1280 for OpenCLIP-G, 4096 for T5-XXL), determining the required text encoder pair. - **Textual Inversion Embeddings:** Detect embedded trigger token vectors and token replacement rules. ### 4. Scheduler & Numerical Solver Reconstruction - **Scheduler Extraction:** Parse `scheduler/scheduler_config.json` parameters: - Number of training timesteps (e.g. 1000). - Beta schedule (`linear`, `scaled_linear`), $\beta_{start}$, $\beta_{end}$. - Prediction type (`epsilon`, `v_prediction`, `flow_prediction`). - Sampling algorithm compatibility (Euler, Heun, DPM-Solver). ### 5. ComfyUI Workflow Graph Inversion - **Embedded PNG/JSON Metadata:** Extract embedded `workflow` and `prompt` JSON metadata stored inside generated PNG `tEXt` chunks or standalone `.json` workflows. - **Execution DAG Inversion:** Reconstruct the node graph connecting: `CheckpointLoader -> CLIPTextEncode (Prompt/Negative) -> KSampler -> VAEDecode -> SaveImage`. --- ## Acceptance Criteria 1. Introduce `analyze_diffusion_pipeline` in REA CLI and MCP contracts, accepting diffusion checkpoints and ComfyUI workflow JSONs. 2. Classify monolithic diffusion checkpoints into UNet, DiT, VAE, and Text Encoder components without loading weights into RAM. 3. Extract cross-attention conditioning dimensions and identify compatible text encoders. 4. Parse scheduler configs and report timesteps, beta schedules, and prediction targets. 5. Invert embedded ComfyUI workflows into structured JSON pipeline graphs. --- ## Related - #3323 [Feature] Inspect GGUF and safetensors model files without loading them - #3665 [Feature] LoRA/PEFT adapter reverse engineering - #3652 [Feature] PyTorch checkpoint reverse engineering

bsakaguchi referenced this issue

2026-10-17 21:01:27 +00:00

[GH-ISSUE #2498] [Perf] Semantic member-mutation traversal takes 84 s and 1.2 GB on pdf.js's 462 KB openjpeg_nowasm_fallback.js #828

bsakaguchi referenced this issue

2026-10-17 21:01:31 +00:00

[GH-ISSUE #2383] [Tracker] Audit analysis and debugging coverage for the TIOBE top 100 languages (October 2026) #794

bsakaguchi referenced this issue

2026-10-17 21:01:57 +00:00

[GH-ISSUE #3670] [Bug] analyze-javascript-application silently skips JavaScript files above ~5 MB (parsed 0, parse_failures 0, no limitation) #1382

bsakaguchi referenced this issue

2026-10-17 21:10:58 +00:00

[PR #1390] [MERGED] fix(javascript): bound member-mutation paths through reassignment loops #2847