Comfy-Org/comfy-kitchen

Fork 0

mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00

[GH-ISSUE #9] INT8 support #5

New issue

Open

opened 2026-10-26 22:19:02 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented

2026-10-26 22:19:02 +00:00

Owner

Originally created by @kanttouchthis on GitHub (Jan 6, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/9

For nvidia gpus with 8.0 <= compute capability < 8.9 which don't have fp8 support, torch._int_mm can be used for torch.int8 matmul. from my own testing, this results in a ~30% performance increase over bf16/fp16 matmul when torch.compiled. this could be used with fp8 by dequantizing before re-quantizing to int8, or it could be used with int8 weights or bit-packed lower precision (u)int weights. in my own extension i use this technique with svd for fast and reasonably accurate 4 bit quantization, but it would be great to see first party support. it should also be fairly easy to implement an int8 matmul kernel in cuda/triton for other gpus.

Originally created by @kanttouchthis on GitHub (Jan 6, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/9 For nvidia gpus with 8.0 <= compute capability < 8.9 which don't have fp8 support, torch._int_mm can be used for torch.int8 matmul. from my own testing, this results in a ~30% performance increase over bf16/fp16 matmul when torch.compiled. this could be used with fp8 by dequantizing before re-quantizing to int8, or it could be used with int8 weights or bit-packed lower precision (u)int weights. in my own [extension](https://github.com/kanttouchthis/ComfyUI-SDNQ) i use this technique with svd for fast and reasonably accurate 4 bit quantization, but it would be great to see first party support. it should also be fairly easy to implement an int8 matmul kernel in cuda/triton for other gpus.

bsakaguchi commented

2026-10-26 22:19:02 +00:00

Author

Owner

@kanttouchthis commented on GitHub (Jan 6, 2026):

here is a code snippet for int8 matmul, which achieved 1.25 it/s compared to 0.91 it/s for bf16/fp16 on flux.1 on my machine (the normal forward pass was also compiled the same way).

@torch.compile(fullgraph=True)
def forward_comfy_cast_weights(self, input):
    if input.numel() // input.shape[-1] <= 16:
        return super().forward_comfy_cast_weights(input)
    
    original_shape = input.shape
    input = input.view((-1, input.shape[-1]))
    dtype = input.dtype
    weight, bias, offload_stream = cast_bias_weight(self, input, offloadable=True)

    scale_x = torch.amax(input.abs(), dim=1, keepdims=True).div_(127)
    x_q = torch.div(input, scale_x).round_().clamp_(-128, 127).to(dtype=torch.int8)

    weight = weight.T
    scale_w = torch.amax(weight.abs(), dim=0, keepdims=True).div_(127)
    weight_q = torch.div(weight, scale_w).round_().clamp_(-128, 127).to(dtype=torch.int8)
    
    output = (torch._int_mm(x_q, weight_q).to(dtype) * scale_x * scale_w)
    output = output.view(*original_shape[:-1], -1)
    
    if bias is not None:
        output += bias
    
    uncast_bias_weight(self, weight, bias, offload_stream)
    return output

@kanttouchthis commented on GitHub (Jan 6, 2026): here is a code snippet for int8 matmul, which achieved 1.25 it/s compared to 0.91 it/s for bf16/fp16 on flux.1 on my machine (the normal forward pass was also compiled the same way). ```python @torch.compile(fullgraph=True) def forward_comfy_cast_weights(self, input): if input.numel() // input.shape[-1] <= 16: return super().forward_comfy_cast_weights(input) original_shape = input.shape input = input.view((-1, input.shape[-1])) dtype = input.dtype weight, bias, offload_stream = cast_bias_weight(self, input, offloadable=True) scale_x = torch.amax(input.abs(), dim=1, keepdims=True).div_(127) x_q = torch.div(input, scale_x).round_().clamp_(-128, 127).to(dtype=torch.int8) weight = weight.T scale_w = torch.amax(weight.abs(), dim=0, keepdims=True).div_(127) weight_q = torch.div(weight, scale_w).round_().clamp_(-128, 127).to(dtype=torch.int8) output = (torch._int_mm(x_q, weight_q).to(dtype) * scale_x * scale_w) output = output.view(*original_shape[:-1], -1) if bias is not None: output += bias uncast_bias_weight(self, weight, bias, offload_stream) return output ```

bsakaguchi commented

2026-10-26 22:19:03 +00:00

Author

Owner

@Ph0rk0z commented on GitHub (Jan 7, 2026):

You check out https://github.com/silveroxides/ComfyUI-QuantOps

It has int8 support without having to do SDNQ. I quantized some TE to int8 and visual performance matched GGUF but with more speed.

I also have been cheating activations with https://github.com/Ph0rk0z/SageAttention2, in the case of FP8 it's faster. Am also kinda topping out at 1.25it/s for 896x1152 chromas/z-images/fluxes

@Ph0rk0z commented on GitHub (Jan 7, 2026): You check out https://github.com/silveroxides/ComfyUI-QuantOps It has int8 support without having to do SDNQ. I quantized some TE to int8 and visual performance matched GGUF but with more speed. I also have been cheating activations with https://github.com/Ph0rk0z/SageAttention2, in the case of FP8 it's faster. Am also kinda topping out at 1.25it/s for 896x1152 chromas/z-images/fluxes

bsakaguchi commented

2026-10-26 22:19:03 +00:00

Author

Owner

@silveroxides commented on GitHub (Mar 9, 2026):

@Ph0rk0z QuantOps now utilize my custom fork of comfy-kitchen:
https://github.com/silveroxides/comfy-kitchen
with prebuild wheels available at:
https://huggingface.co/silveroxides/Chroma1-HD-fp8-scaled/tree/main/experimental

If combined with this PR (Because annoyingly ComfyUI disables triton on comfy-kitchen):
https://github.com/Comfy-Org/ComfyUI/pull/12730

Then you can run models using tensorwise int8 with matmul faster than ComfyUI core fp8 even.
This should be supported on 30,40 and 50-series cards.

@silveroxides commented on GitHub (Mar 9, 2026): @Ph0rk0z QuantOps now utilize my custom fork of comfy-kitchen: https://github.com/silveroxides/comfy-kitchen with prebuild wheels available at: https://huggingface.co/silveroxides/Chroma1-HD-fp8-scaled/tree/main/experimental If combined with this PR (Because annoyingly ComfyUI disables triton on comfy-kitchen): https://github.com/Comfy-Org/ComfyUI/pull/12730 Then you can run models using tensorwise int8 with matmul faster than ComfyUI core fp8 even. This should be supported on 30,40 and 50-series cards.

bsakaguchi commented

2026-10-26 22:19:03 +00:00

Author

Owner

@Ph0rk0z commented on GitHub (Mar 9, 2026):

Yup, I have been hacking branches of your kitchen to get rid of unsupported FP8 functions in cuda and force-enabling triton. Its very hard to choose which branch though. I have to guestimate by commits. Also made it run on cuda 12.6 and bypassed those checks. So far model goes:https://github.com/BobJohnson24/ComfyUI-INT8-Fast and TE is off your plugin.

Whether it's faster than compiled FP8 with patched triton is hit or miss but I think int8 quality is slightly better and less prone to breaking when using mixed FP8 quants.

@Ph0rk0z commented on GitHub (Mar 9, 2026): Yup, I have been hacking branches of your kitchen to get rid of unsupported FP8 functions in cuda and force-enabling triton. Its very hard to choose which branch though. I have to guestimate by commits. Also made it run on cuda 12.6 and bypassed those checks. So far model goes:https://github.com/BobJohnson24/ComfyUI-INT8-Fast and TE is off your plugin. Whether it's faster than compiled FP8 with patched triton is hit or miss but I think int8 quality is slightly better and less prone to breaking when using mixed FP8 quants.

bsakaguchi commented

2026-10-26 22:19:03 +00:00

Author

Owner

@silveroxides commented on GitHub (Mar 10, 2026):

@Ph0rk0z
it is main now and pre-built wheels exist
https://github.com/Comfy-Org/ComfyUI/issues/12830#issuecomment-4030593734

@silveroxides commented on GitHub (Mar 10, 2026): @Ph0rk0z it is main now and pre-built wheels exist https://github.com/Comfy-Org/ComfyUI/issues/12830#issuecomment-4030593734

bsakaguchi commented

2026-10-26 22:19:03 +00:00

Author

Owner

@Ph0rk0z commented on GitHub (Mar 11, 2026):

Main seems to be working. Outputs are a little different from the old int8 branch. Not sure if for better or for worse. I think the TE is better but I've only used the int8-fast for chroma itself. I'll double check it works with models too, last time it didn't work with tensworwise.

@Ph0rk0z commented on GitHub (Mar 11, 2026): Main seems to be working. Outputs are a little different from the old int8 branch. Not sure if for better or for worse. I think the TE is better but I've only used the int8-fast for chroma itself. I'll double check it works with models too, last time it didn't work with tensworwise.

bsakaguchi commented

2026-10-26 22:19:03 +00:00

Author

Owner

@silveroxides commented on GitHub (Jun 2, 2026):

My PR is finally up. https://github.com/Comfy-Org/comfy-kitchen/pull/37

@silveroxides commented on GitHub (Jun 2, 2026): My PR is finally up. https://github.com/Comfy-Org/comfy-kitchen/pull/37

bsakaguchi referenced this issue

2026-10-26 22:20:19 +00:00

[PR #37] [MERGED] Add support for int8 quantization backend #94