[GH-ISSUE #22] Bundle torch-cublas-hgemm #14

New issue

Closed

opened 2026-10-26 22:19:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 22:19:08 +00:00

Owner

Originally created by @woct0rdho on GitHub (Jan 17, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/22

As you're already using cublas, would you consider bundling torch-cublas-hgemm in this repo? It's been in ComfyUI's --fast for a while. Bundling it would make it easier for more people to install it.

Originally created by @woct0rdho on GitHub (Jan 17, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/22 As you're already using cublas, would you consider bundling [torch-cublas-hgemm](https://github.com/aredden/torch-cublas-hgemm) in this repo? It's been in ComfyUI's `--fast` for a while. Bundling it would make it easier for more people to install it.

bsakaguchi closed this issue

2026-10-26 22:19:08 +00:00

bsakaguchi commented

2026-10-26 22:19:09 +00:00

Author

Owner

@contentis commented on GitHub (Jan 22, 2026):

Can you explain what the added gain would be? PyTorch is already relying on cublasLt for all GEMM operations and it should do the exact same calls. ReLU, GeLU should be possible to be enforced but the seed-up not very large.

If I'm not mistaken you can enable the half accum in PyTorch directly by using https://docs.pytorch.org/docs/stable/backends.html#torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction
And I belive this is even a start-up flag in ComfyUI

@contentis commented on GitHub (Jan 22, 2026): Can you explain what the added gain would be? PyTorch is already relying on cublasLt for all GEMM operations and it should do the exact same calls. ReLU, GeLU should be possible to be enforced but the seed-up not very large. If I'm not mistaken you can enable the half accum in PyTorch directly by using https://docs.pytorch.org/docs/stable/backends.html#torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction And I belive this is even a start-up flag in ComfyUI

bsakaguchi commented

2026-10-26 22:19:09 +00:00

Author

Owner

@woct0rdho commented on GitHub (Jan 22, 2026):

In my impression --fast cublas_ops indeed gives speedup when running SDXL, even when --fast fp16_accumulation is already enabled. But I may need to do benchmarking again when I have time.

@woct0rdho commented on GitHub (Jan 22, 2026): In my impression `--fast cublas_ops` indeed gives speedup when running SDXL, even when `--fast fp16_accumulation` is already enabled. But I may need to do benchmarking again when I have time.

bsakaguchi commented

2026-10-26 22:19:09 +00:00

Author

Owner

@woct0rdho commented on GitHub (Jan 26, 2026):

Nevermind. After doing benchmarks again and looking at the source code, I can confirm that nowadays torch.nn.Linear is as fast as CublasLinear when fp16 accum is enabled.

@woct0rdho commented on GitHub (Jan 26, 2026): Nevermind. After doing benchmarks again and looking at the source code, I can confirm that nowadays `torch.nn.Linear` is as fast as `CublasLinear` when fp16 accum is enabled.

bsakaguchi referenced this issue

2026-10-26 22:20:12 +00:00

[PR #14] [CLOSED] Try to fix windows build. #82