Comfy-Org/comfy-kitchen

Fork 0

mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00

[GH-ISSUE #27] Clarification regarding the readme example #13

New issue

Open

opened 2026-10-26 22:19:08 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 22:19:08 +00:00

Owner

Originally created by @sayakpaul on GitHub (Jan 29, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/27

from comfy_kitchen.tensor import QuantizedTensor, TensorCoreFP8Layout, TensorCoreNVFP4Layout

# Quantize a tensor
x = torch.randn(128, 256, device="cuda", dtype=torch.bfloat16)
qt = QuantizedTensor.from_float(x, TensorCoreFP8Layout)

# Operations dispatch to optimized kernels automatically
output = torch.nn.functional.linear(qt, weight_qt)

# Dequantize back to float
dq = qt.dequantize()

How is weight_qt computed in this case?

Also, is it recommended to quantize the input (which seems to be qt in this case) instead of the weights?

Originally created by @sayakpaul on GitHub (Jan 29, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/27 ```py from comfy_kitchen.tensor import QuantizedTensor, TensorCoreFP8Layout, TensorCoreNVFP4Layout # Quantize a tensor x = torch.randn(128, 256, device="cuda", dtype=torch.bfloat16) qt = QuantizedTensor.from_float(x, TensorCoreFP8Layout) # Operations dispatch to optimized kernels automatically output = torch.nn.functional.linear(qt, weight_qt) # Dequantize back to float dq = qt.dequantize() ``` How is `weight_qt` computed in this case? Also, is it recommended to quantize the input (which seems to be `qt` in this case) instead of the weights?

bsakaguchi commented

2026-10-26 22:19:08 +00:00

Author

Owner

@contentis commented on GitHub (Feb 2, 2026):

The from_float function calls into the quantize function from the provided LayoutCls. In this case it would call this

How is weight_qt computed in this case?

Here we assume that weight_qt is already an instance of QuantizedTensor. Usually we have a checkpoint that already has the weights quantized and load them as such directly by creating an instance. If you have a high-precision tensor/weight you could also leverage from_float to quantize it.

Also, is it recommended to quantize the input (which seems to be qt in this case) instead of the weights?

This entirely depends on your goal and dosen't have a short answer. The TLDR is that we can distinguish between Weight-Only (WoQ) and Weigth+Activation. When using WoQ methods we only quantize the weights and keep the input in high-precision float. When runnning an operation (e.g. MatMul) we upcast the weight into an intermediate buffer and perform the math in high-precision. out = x @ weight_qt.dequantize()
This saved VRAM but has no performance impact (usually a bit slower than high precision) but its easy to do.

For the other we can make use of dedicated Hardware like FP8 Tensor Cores. And this is where we see the performance gains, although often a bit harder to get good quality out of the box.

Here is some more in-depth explaination if you're interested: https://developer.nvidia.com/blog/model-quantization-concepts-methods-and-why-it-matters/

@contentis commented on GitHub (Feb 2, 2026): The `from_float` function calls into the `quantize` [function](https://github.com/Comfy-Org/comfy-kitchen/blob/main/comfy_kitchen/tensor/base.py#L234C9-L234C14) from the provided LayoutCls. In this case it would call [this](https://github.com/Comfy-Org/comfy-kitchen/blob/main/comfy_kitchen/tensor/fp8.py#L46) >How is weight_qt computed in this case? Here we assume that `weight_qt` is already an instance of `QuantizedTensor`. Usually we have a checkpoint that already has the weights quantized and load them as such directly by creating an [instance](https://github.com/Comfy-Org/comfy-kitchen/blob/main/comfy_kitchen/tensor/base.py#L166). If you have a high-precision tensor/weight you could also leverage `from_float` to quantize it. >Also, is it recommended to quantize the input (which seems to be qt in this case) instead of the weights? This entirely depends on your goal and dosen't have a short answer. The TLDR is that we can distinguish between Weight-Only (WoQ) and Weigth+Activation. When using WoQ methods we only quantize the weights and keep the input in high-precision float. When runnning an operation (e.g. MatMul) we upcast the weight into an intermediate buffer and perform the math in high-precision. `out = x @ weight_qt.dequantize()` This saved VRAM but has no performance impact (usually a bit slower than high precision) but its easy to do. For the other we can make use of dedicated Hardware like FP8 Tensor Cores. And this is where we see the performance gains, although often a bit harder to get good quality out of the box. Here is some more in-depth explaination if you're interested: https://developer.nvidia.com/blog/model-quantization-concepts-methods-and-why-it-matters/

bsakaguchi commented

2026-10-26 22:19:09 +00:00

Author

Owner

@sayakpaul commented on GitHub (Feb 2, 2026):

Thanks so much!

For the other we can make use of dedicated Hardware like FP8 Tensor Cores. And this is where we see the performance gains, although often a bit harder to get good quality out of the box.

Oh? I was under the impression that even for weight-only FP8, there can still be kernels written to utilize the dedicated FP8 tensor cores. I guess you meant a workflow like so?

Keep params in FP8, casts FP32/FP16/BF16 activations to FP8 on-the-fly
Use FP8 tensor cores for the actual computation

# Pseudo-code for optimized weight-only FP8
out = fp8_matmul(
    x.cast_to_fp8(),           # Cast activation to FP8 on-the-fly
    weight_fp8,                 # Already stored in FP8
    output_dtype=fp16           # Accumulate in higher precision
)

Here we assume that weight_qt is already an instance of QuantizedTensor. Usually we have a checkpoint that already has the weights quantized and load them as such directly by creating an instance. If you have a high-precision tensor/weight you could also leverage from_float to quantize it.

That's what I had assumed but having a fully complete example (potentially with a checkpoint) would have been great as a starting point I believe.

@sayakpaul commented on GitHub (Feb 2, 2026): Thanks so much! > For the other we can make use of dedicated Hardware like FP8 Tensor Cores. And this is where we see the performance gains, although often a bit harder to get good quality out of the box. Oh? I was under the impression that even for weight-only FP8, there can still be kernels written to utilize the dedicated FP8 tensor cores. I guess you meant a workflow like so? * Keep params in FP8, casts FP32/FP16/BF16 activations to FP8 on-the-fly * Use FP8 tensor cores for the actual computation ```py # Pseudo-code for optimized weight-only FP8 out = fp8_matmul( x.cast_to_fp8(), # Cast activation to FP8 on-the-fly weight_fp8, # Already stored in FP8 output_dtype=fp16 # Accumulate in higher precision ) ``` > Here we assume that weight_qt is already an instance of QuantizedTensor. Usually we have a checkpoint that already has the weights quantized and load them as such directly by creating an [instance](https://github.com/Comfy-Org/comfy-kitchen/blob/main/comfy_kitchen/tensor/base.py#L166). If you have a high-precision tensor/weight you could also leverage from_float to quantize it. That's what I had assumed but having a fully complete example (potentially with a checkpoint) would have been great as a starting point I believe.

bsakaguchi commented

2026-10-26 22:19:09 +00:00

Author

Owner

@contentis commented on GitHub (Feb 2, 2026):

Yeah, documentation and tooling could be improved and is something to be looked at. One last minor note, this:x.cast_to_fp8() is not a cast operation in the classcial sense but a scaled transformation where the tensor is first resaled to match the lower-precisions dynamic range and then catsted (round nearest).

@contentis commented on GitHub (Feb 2, 2026): Yeah, documentation and tooling could be improved and is something to be looked at. One last minor note, this:`x.cast_to_fp8()` is not a cast operation in the classcial sense but a scaled transformation where the tensor is first resaled to match the lower-precisions dynamic range and then catsted (round nearest).

bsakaguchi commented

2026-10-26 22:19:09 +00:00

Author

Owner

@sayakpaul commented on GitHub (Feb 2, 2026):

cast_to_fp8() is not a cast operation in the classcial sense but a scaled transformation where the tensor is first resaled to match the lower-precisions dynamic range and then catsted (round nearest).

Yes, of course. Lots of configurations there two in terms of how much granularity one wants (per-row vs. per-tensor, etc.).

Yeah, documentation and tooling could be improved and is something to be looked at.

Maybe we could keep this thread open till then?

@sayakpaul commented on GitHub (Feb 2, 2026): > cast_to_fp8() is not a cast operation in the classcial sense but a scaled transformation where the tensor is first resaled to match the lower-precisions dynamic range and then catsted (round nearest). Yes, of course. Lots of configurations there two in terms of how much granularity one wants (per-row vs. per-tensor, etc.). > Yeah, documentation and tooling could be improved and is something to be looked at. Maybe we could keep this thread open till then?

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/comfy-kitchen#13

No description provided.

Rows
Columns