Comfy-Org/comfy-quants

Fork 0

mirror of https://github.com/Comfy-Org/comfy-quants.git synced 2026-11-04 23:37:27 +00:00

Ideogram 4.0 adapters and the stock-ComfyUI int8_tensorwise format #9

New issue

Closed

opened 2026-10-26 22:28:59 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 22:28:59 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/Comfy-Org/comfy-quants/pull/6
Author: @HK416-TYPED
Created: 6/26/2026
Status: ✅ Merged
Merged: 7/3/2026
Merged by: @alexisrolland

Base: main ← Head: main

📝 Commits (10+)

67b4e30 Add shared stock-DiT contract framework for new model families
c6e5237 Add FLUX.1 model-family adapter (FP8/MXFP8/NVFP4)
88620b4 Add FLUX.2 model-family adapter with mixed-precision recipe (FP8/MXFP8/NVFP4)
39c00b8 Add LTX-Video model-family adapter (FP8/MXFP8/NVFP4)
fc5ed61 Add Ideogram 4.0 model-family adapter (FP8/MXFP8/NVFP4)
fc6f3e0 docs: list flux/flux2/ltxv/ideogram4 in the doc indexes
16c6449 Address review: fix nvfp4-mixed size note and the bf16 transcode hop
ff6263f Preserve source safetensors metadata and the qwen 2511 sentinel in all whole-checkpoint writers
2b30d74 Add LTX-2 (LTX-2.3) model-family adapter and stock-ComfyUI int8_tensorwise format
6b18620 Address review: dedupe writer metadata/sentinel logic, trim configs, clarify docs

📊 Changes

77 files changed (+4504 additions, -26 deletions)

View changed files

➕ configs/flux2_fp8.yaml (+34 -0)
➕ configs/flux2_fp8_e5m2.yaml (+34 -0)
➕ configs/flux2_fp8_mixed.yaml (+39 -0)
➕ configs/flux2_int8_tensorwise.yaml (+34 -0)
➕ configs/flux2_mxfp8.yaml (+34 -0)
➕ configs/flux2_mxfp8_mixed.yaml (+39 -0)
➕ configs/flux2_nvfp4.yaml (+34 -0)
➕ configs/flux2_nvfp4_mixed.yaml (+40 -0)
➕ configs/flux_fp8.yaml (+34 -0)
➕ configs/flux_fp8_e5m2.yaml (+34 -0)
➕ configs/flux_mxfp8.yaml (+34 -0)
➕ configs/flux_nvfp4.yaml (+34 -0)
➕ configs/ideogram4_fp8.yaml (+33 -0)
➕ configs/ideogram4_fp8_e5m2.yaml (+33 -0)
➕ configs/ideogram4_mxfp8.yaml (+33 -0)
➕ configs/ideogram4_nvfp4.yaml (+33 -0)
➕ configs/ltx2_fp8.yaml (+38 -0)
➕ configs/ltx2_fp8_e5m2.yaml (+38 -0)
➕ configs/ltx2_int8_tensorwise.yaml (+39 -0)
➕ configs/ltx2_int8_w8a8.yaml (+39 -0)

...and 57 more files

📄 Description

Adds four new stock-ComfyUI-native model families on top of the existing Qwen/Anima support:
FLUX.1, FLUX.2, LTX-Video (LTXV), and Ideogram 4.0. Each reuses the existing
FP8 / MXFP8 / NVFP4 export pipeline — only the per-family adapters/contracts are new.

What's added

Shared stock_dit_contract framework — generic block-group contract + graph builder
supporting N repeated block groups (e.g. FLUX's double + single streams).
FLUX.1 (MMDiT, 304 quantizable Linears) · FLUX.2 (gated MLP + global modulation, 160;
plus an official-fp8mixed-aligned mixed-precision recipe that keeps attention bf16 → 128) ·
LTX-Video (Lightricks video DiT, 280; released checkpoint uses a model.diffusion_model.
key prefix) · Ideogram 4.0 (open-weight NextDiT/Lumina2, 204).
Per-family configs (FP8 e4m3/e5m2, MXFP8, NVFP4) + contract unit tests.
Docs: docs/quantization/{flux,flux2,ltxv,ideogram4}.md + index updates.
Ideogram 4.0 doc also covers the fp8→bf16→mxfp8 transcoding path (no public dense source).

Appended: LTX-2 (LTX-2.3) family + stock-ComfyUI `int8_tensorwise` format

ltx2 adapter — LTX-2.3 22B audio-video DiT (48 blocks × 34 Linears incl. the 2.3
gated-attention to_gate_logits); selection reproduces the official Lightricks recipe
(blocks 2–45 → 1,496 layers, string-identical to the official fp8/nvfp4 releases);
single-file bundle passthrough (vae/audio_vae/vocoder/text_embedding_projection).
int8_tensorwise — new format/algorithm/writer/CLI vertical producing checkpoints for
stock ComfyUI ≥ 0.27 QUANT_ALGOS["int8_tensorwise"] (SM ≥ 7.5): int8 weight +
fp32 [out,1] scale + ConvRot (regular Hadamard, group 256); marker byte-equal to the
stock save path; quant math bit-faithful to comfy-kitchen ≥ 0.2.15. Family configs for
ltx2 / flux2 / qwen trio, selection locked to each family's official recipe by test.
Writer fixes (all whole-checkpoint writers): preserve source __metadata__
(stock ComfyUI builds the LTX-2 architecture from __metadata__["config"]) and emit the
qwen-2511 __index_timestep_zero__ sentinel beyond the FP8 writer.

Validation

Each adapter's quantized-tensor contract validated 100% (name + shape) against the real
published checkpoint headers — FLUX.1 304/304, FLUX.2 160/160, LTXV 280/280, Ideogram4 204/204,
LTX-2.3 1,632/1,632 (selection 1,496/1,496 vs the official quantized releases).
Full unit suite green (269 passed, 4 skipped), incl. an external comfy-kitchen parity
test for the int8 quant math (48/48 bit-exact vs comfy-kitchen 0.2.16).
Real export + ComfyUI sampling on Blackwell (torch 2.10, all ops native incl. mxfp8):
LTX-2.3 int8_tensorwise = 45.4 dB video PSNR vs bf16, 1.15× sampling speed, −28% VRAM;
FLUX.2 int8_tensorwise = 32.6 dB (vs 20.1 dB for fp8 on this distilled model).

Commits are split per family (framework → flux → flux2 → ltxv → ideogram4 → docs →
writer fixes → ltx2 + int8_tensorwise).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/Comfy-Org/comfy-quants/pull/6 **Author:** [@HK416-TYPED](https://github.com/HK416-TYPED) **Created:** 6/26/2026 **Status:** ✅ Merged **Merged:** 7/3/2026 **Merged by:** [@alexisrolland](https://github.com/alexisrolland) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (10+) - [`67b4e30`](https://github.com/Comfy-Org/comfy-quants/commit/67b4e30e259203173e1ce76b0dbe9245baa9ec40) Add shared stock-DiT contract framework for new model families - [`c6e5237`](https://github.com/Comfy-Org/comfy-quants/commit/c6e5237d49bc0daff99d11cf2ec79aefb2e13f6a) Add FLUX.1 model-family adapter (FP8/MXFP8/NVFP4) - [`88620b4`](https://github.com/Comfy-Org/comfy-quants/commit/88620b4054e4587e130960b926e23482988fc0a5) Add FLUX.2 model-family adapter with mixed-precision recipe (FP8/MXFP8/NVFP4) - [`39c00b8`](https://github.com/Comfy-Org/comfy-quants/commit/39c00b8cbe6de097a161063457b4cf50c826ce45) Add LTX-Video model-family adapter (FP8/MXFP8/NVFP4) - [`fc5ed61`](https://github.com/Comfy-Org/comfy-quants/commit/fc5ed617daceb26fcb80ccef7ad8c66eda0e52fa) Add Ideogram 4.0 model-family adapter (FP8/MXFP8/NVFP4) - [`fc6f3e0`](https://github.com/Comfy-Org/comfy-quants/commit/fc6f3e094da204bb117f28bb76d8949b2e33e59a) docs: list flux/flux2/ltxv/ideogram4 in the doc indexes - [`16c6449`](https://github.com/Comfy-Org/comfy-quants/commit/16c6449d862ca146ed1e8420fda69b2808ac01dc) Address review: fix nvfp4-mixed size note and the bf16 transcode hop - [`ff6263f`](https://github.com/Comfy-Org/comfy-quants/commit/ff6263f3102d54718da2ca6a6789aa9860210b14) Preserve source safetensors metadata and the qwen 2511 sentinel in all whole-checkpoint writers - [`2b30d74`](https://github.com/Comfy-Org/comfy-quants/commit/2b30d749fae9b62ae46d77abd1219db8099d1e8a) Add LTX-2 (LTX-2.3) model-family adapter and stock-ComfyUI int8_tensorwise format - [`6b18620`](https://github.com/Comfy-Org/comfy-quants/commit/6b186203ccffcd9d34f7e248688f19fc32acfc1e) Address review: dedupe writer metadata/sentinel logic, trim configs, clarify docs ### 📊 Changes **77 files changed** (+4504 additions, -26 deletions) <details> <summary>View changed files</summary> ➕ `configs/flux2_fp8.yaml` (+34 -0) ➕ `configs/flux2_fp8_e5m2.yaml` (+34 -0) ➕ `configs/flux2_fp8_mixed.yaml` (+39 -0) ➕ `configs/flux2_int8_tensorwise.yaml` (+34 -0) ➕ `configs/flux2_mxfp8.yaml` (+34 -0) ➕ `configs/flux2_mxfp8_mixed.yaml` (+39 -0) ➕ `configs/flux2_nvfp4.yaml` (+34 -0) ➕ `configs/flux2_nvfp4_mixed.yaml` (+40 -0) ➕ `configs/flux_fp8.yaml` (+34 -0) ➕ `configs/flux_fp8_e5m2.yaml` (+34 -0) ➕ `configs/flux_mxfp8.yaml` (+34 -0) ➕ `configs/flux_nvfp4.yaml` (+34 -0) ➕ `configs/ideogram4_fp8.yaml` (+33 -0) ➕ `configs/ideogram4_fp8_e5m2.yaml` (+33 -0) ➕ `configs/ideogram4_mxfp8.yaml` (+33 -0) ➕ `configs/ideogram4_nvfp4.yaml` (+33 -0) ➕ `configs/ltx2_fp8.yaml` (+38 -0) ➕ `configs/ltx2_fp8_e5m2.yaml` (+38 -0) ➕ `configs/ltx2_int8_tensorwise.yaml` (+39 -0) ➕ `configs/ltx2_int8_w8a8.yaml` (+39 -0) _...and 57 more files_ </details> ### 📄 Description Adds four new stock-ComfyUI-native model families on top of the existing Qwen/Anima support: **FLUX.1**, **FLUX.2**, **LTX-Video (LTXV)**, and **Ideogram 4.0**. Each reuses the existing FP8 / MXFP8 / NVFP4 export pipeline — only the per-family adapters/contracts are new. ## What's added - **Shared `stock_dit_contract` framework** — generic block-group contract + graph builder supporting N repeated block groups (e.g. FLUX's double + single streams). - **FLUX.1** (MMDiT, 304 quantizable Linears) · **FLUX.2** (gated MLP + global modulation, 160; plus an official-`fp8mixed`-aligned mixed-precision recipe that keeps attention bf16 → 128) · **LTX-Video** (Lightricks video DiT, 280; released checkpoint uses a `model.diffusion_model.` key prefix) · **Ideogram 4.0** (open-weight NextDiT/Lumina2, 204). - Per-family configs (FP8 e4m3/e5m2, MXFP8, NVFP4) + contract unit tests. - Docs: `docs/quantization/{flux,flux2,ltxv,ideogram4}.md` + index updates. - Ideogram 4.0 doc also covers the fp8→bf16→mxfp8 **transcoding** path (no public dense source). ## Appended: LTX-2 (LTX-2.3) family + stock-ComfyUI `int8_tensorwise` format - **`ltx2` adapter** — LTX-2.3 22B audio-video DiT (48 blocks × 34 Linears incl. the 2.3 gated-attention `to_gate_logits`); selection reproduces the official Lightricks recipe (blocks 2–45 → **1,496 layers, string-identical to the official fp8/nvfp4 releases**); single-file bundle passthrough (vae/audio_vae/vocoder/text_embedding_projection). - **`int8_tensorwise`** — new format/algorithm/writer/CLI vertical producing checkpoints for stock ComfyUI ≥ 0.27 `QUANT_ALGOS["int8_tensorwise"]` (SM ≥ 7.5): int8 weight + fp32 `[out,1]` scale + ConvRot (regular Hadamard, group 256); marker byte-equal to the stock save path; quant math bit-faithful to comfy-kitchen ≥ 0.2.15. Family configs for ltx2 / flux2 / qwen trio, selection locked to each family's official recipe by test. - **Writer fixes (all whole-checkpoint writers)**: preserve source `__metadata__` (stock ComfyUI builds the LTX-2 architecture from `__metadata__["config"]`) and emit the qwen-2511 `__index_timestep_zero__` sentinel beyond the FP8 writer. ## Validation - Each adapter's quantized-tensor contract validated **100% (name + shape)** against the real published checkpoint headers — FLUX.1 304/304, FLUX.2 160/160, LTXV 280/280, Ideogram4 204/204, **LTX-2.3 1,632/1,632** (selection 1,496/1,496 vs the official quantized releases). - Full unit suite green (**269 passed, 4 skipped**), incl. an external comfy-kitchen parity test for the int8 quant math (48/48 bit-exact vs comfy-kitchen 0.2.16). - Real export + ComfyUI sampling on Blackwell (torch 2.10, all ops native incl. mxfp8): LTX-2.3 int8_tensorwise = 45.4 dB video PSNR vs bf16, 1.15× sampling speed, −28% VRAM; FLUX.2 int8_tensorwise = 32.6 dB (vs 20.1 dB for fp8 on this distilled model). Commits are split per family (framework → flux → flux2 → ltxv → ideogram4 → docs → writer fixes → ltx2 + int8_tensorwise). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 22:28:59 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-quants#9

No description provided.

Rows
Columns