Comfy-Org/comfy-kitchen

Fork 0

mirror of https://github.com/Comfy-Org/comfy-kitchen.git synced 2026-11-04 23:37:42 +00:00

[GH-ISSUE #98] CUTLASS INT8 dequant selects poor config for tall MiniMax H3 shapes on RTX A6000 #30

New issue

Open

opened 2026-10-26 22:19:22 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 22:19:22 +00:00

Owner

Originally created by @Many0therFunctions on GitHub (Aug 7, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/98

Summary

The automatic cutlass_int8_dequant dispatch appears to select a poor CUTLASS configuration for tall sequence shapes on an NVIDIA RTX A6000 (sm86). The exact MiniMax H3 projection shapes from ComfyUI logs reach the CUTLASS path, but QKV is nearly no faster than BF16 until a faster compiled configuration is selected explicitly.

Environment:

GPU: NVIDIA RTX A6000, compute capability sm86, 48 GB
PyTorch: 2.13.0+cu132
CUDA runtime: 13.2
comfy-kitchen: 0.2.26
dtype: bfloat16 output
ConvRot: true, groupsize 256
M = 80661 tokens

Reproduction

The four production projection shapes are:

M=80661 N=21504 K=5376  # QKV
M=80661 N=5376  K=7168  # attention output
M=80661 N=28672 K=5376  # MLP up
M=80661 N=5376  K=14336 # MLP down + fused SwiGLU

ComfyUI logs show the normal path:

[ComfyUI INT8 dispatch] path=int8_quantized_tensor input=(80661, 5376) weight=(21504, 5376) quant=int8_tensorwise convrot=True
[comfy-kitchen INT8] M=80661 N=21504 K=5376 convrot=True input_act=None path=cutlass

Production timings for one projection call on the A6000:

QKV BF16:          148.8 ms
QKV INT8 ConvRot:  144.3 ms  (1.03x)

The extension exposes benchmark_cutlass_int8_dequant_config. Sweeping the available configs for the same QKV tensors produced:

config 0: ~84.0 ms kernel-only
config 3: ~98.0 ms
config 2: ~136.5 ms
config 7: ~136.6 ms
config 1: ~137.2 ms
config 12: ~138.0 ms
config 13: ~139.1 ms

An end-to-end test using the existing ConvRot activation quantizer plus explicit CUTLASS config 0 measured approximately 89.5 ms, versus 144.3 ms through the normal automatic call.

For the same M, the fastest configurations were:

QKV       N=21504 K=5376:  config 0 (~84.0 ms)
Attn out  N=5376  K=7168:  config 0 (~26.7 ms)
MLP up   N=28672 K=5376:  config 0 (~111.7 ms)
MLP down N=5376  K=14336: config 13 (~54.5 ms), config 0 ~55.0 ms

Expected behavior

The automatic CUTLASS selection should benchmark or select a configuration based on the actual (M, N, K, device) geometry, or use a shape heuristic that handles tall sequence lengths. A fixed/default configuration is especially problematic because M is the packed MiniMax sequence length and varies with video duration, spatial resolution, audio length, text, and reference media.

The QKV and MLP-up shapes are architecture-fixed in N/K, but M changes with the workflow's packed sequence length. The issue is therefore not specific to one literal M=80661; the dispatcher should account for tall M and wide N.

Could you expose the chosen CUTLASS config in diagnostics and add a benchmark/heuristic path for these tall Ampere shapes? The existing benchmark binding made the discrepancy straightforward to verify.

The full benchmark script and logs can be provided if useful.

Originally created by @Many0therFunctions on GitHub (Aug 7, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/98 ## Summary The automatic `cutlass_int8_dequant` dispatch appears to select a poor CUTLASS configuration for tall sequence shapes on an NVIDIA RTX A6000 (sm86). The exact MiniMax H3 projection shapes from ComfyUI logs reach the CUTLASS path, but QKV is nearly no faster than BF16 until a faster compiled configuration is selected explicitly. Environment: - GPU: NVIDIA RTX A6000, compute capability sm86, 48 GB - PyTorch: 2.13.0+cu132 - CUDA runtime: 13.2 - comfy-kitchen: 0.2.26 - dtype: bfloat16 output - ConvRot: true, groupsize 256 - `M = 80661` tokens ## Reproduction The four production projection shapes are: ```text M=80661 N=21504 K=5376 # QKV M=80661 N=5376 K=7168 # attention output M=80661 N=28672 K=5376 # MLP up M=80661 N=5376 K=14336 # MLP down + fused SwiGLU ``` ComfyUI logs show the normal path: ```text [ComfyUI INT8 dispatch] path=int8_quantized_tensor input=(80661, 5376) weight=(21504, 5376) quant=int8_tensorwise convrot=True [comfy-kitchen INT8] M=80661 N=21504 K=5376 convrot=True input_act=None path=cutlass ``` Production timings for one projection call on the A6000: ```text QKV BF16: 148.8 ms QKV INT8 ConvRot: 144.3 ms (1.03x) ``` The extension exposes `benchmark_cutlass_int8_dequant_config`. Sweeping the available configs for the same QKV tensors produced: ```text config 0: ~84.0 ms kernel-only config 3: ~98.0 ms config 2: ~136.5 ms config 7: ~136.6 ms config 1: ~137.2 ms config 12: ~138.0 ms config 13: ~139.1 ms ``` An end-to-end test using the existing ConvRot activation quantizer plus explicit CUTLASS config 0 measured approximately `89.5 ms`, versus `144.3 ms` through the normal automatic call. For the same `M`, the fastest configurations were: ```text QKV N=21504 K=5376: config 0 (~84.0 ms) Attn out N=5376 K=7168: config 0 (~26.7 ms) MLP up N=28672 K=5376: config 0 (~111.7 ms) MLP down N=5376 K=14336: config 13 (~54.5 ms), config 0 ~55.0 ms ``` ## Expected behavior The automatic CUTLASS selection should benchmark or select a configuration based on the actual `(M, N, K, device)` geometry, or use a shape heuristic that handles tall sequence lengths. A fixed/default configuration is especially problematic because `M` is the packed MiniMax sequence length and varies with video duration, spatial resolution, audio length, text, and reference media. The QKV and MLP-up shapes are architecture-fixed in `N/K`, but `M` changes with the workflow's packed sequence length. The issue is therefore not specific to one literal `M=80661`; the dispatcher should account for tall `M` and wide `N`. Could you expose the chosen CUTLASS config in diagnostics and add a benchmark/heuristic path for these tall Ampere shapes? The existing benchmark binding made the discrepancy straightforward to verify. The full benchmark script and logs can be provided if useful.

bsakaguchi commented

2026-10-26 22:19:23 +00:00

Author

Owner

@coderabbitai[bot] commented on GitHub (Aug 7, 2026):

🔗 Related PRs

Comfy-Org/comfy-kitchen#88 - Add swiglu input_act and partial-rotary rot_dim for rms_rope [merged]
Comfy-Org/comfy-kitchen#90 - Add optimized w4a8 with int8 codebook [merged]
Comfy-Org/comfy-kitchen#94 - HIP: fix RDNA4 GEMM tile selection, vectorize fp8 elementwise kernels [open]
Comfy-Org/comfy-kitchen#96 - a4w8: use frozen codebook LUT + row-chunked quantize, improve LoRA application [merged]

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

@coderabbitai[bot] commented on GitHub (Aug 7, 2026):  <details> <summary>🔗 Related PRs</summary> Comfy-Org/comfy-kitchen#88 - Add swiglu input_act and partial-rotary rot_dim for rms_rope [merged] Comfy-Org/comfy-kitchen#90 - Add optimized w4a8 with int8 codebook [merged] Comfy-Org/comfy-kitchen#94 - HIP: fix RDNA4 GEMM tile selection, vectorize fp8 elementwise kernels [open] Comfy-Org/comfy-kitchen#96 - a4w8: use frozen codebook LUT + row-chunked quantize, improve LoRA application [merged] </details> --- <details> <summary> 🧪 Issue enrichment is currently in open beta.</summary> You can configure auto-planning by selecting labels in the issue_enrichment configuration. To disable automatic issue enrichment, add the following to your `.coderabbit.yaml`: ```yaml issue_enrichment: auto_enrich: enabled: false ``` </details> 💬 Have feedback or questions? Drop into our [discord](https://discord.gg/coderabbit)!

bsakaguchi commented

2026-10-26 22:19:23 +00:00

Author

Owner

@Many0therFunctions commented on GitHub (Aug 7, 2026):

Additional sm86 tuning results

I completed a resumable sweep on an RTX A6000 (sm86), covering:

M = 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 49152, 65536, 80661, 98304
all four MiniMax H3 projection geometries
all 14 compiled CUTLASS configurations
3 randomized passes per configuration

The relevant configurations are:

config 0: 128x256x64, ordinary identity scheduling
config 13: 128x256x64, Lean Stream-K scheduling

The stable sm86 result is:

M >= 12288, N >= 16384, 4096 <= K <= 8192
=> config 0 is consistently preferable to config 13

I then validated neighboring geometries:

N: 16384, 20480, 21248, 21504, 21760, 24576, 28672
K: 4096, 5120, 5376, 6144, 7168, 8192
M: 8192, 12288, 16384, 32768

For M >= 12288, config 0 beat config 13 by approximately 1.62x-1.76x across the tested region. At M=8192, config 0 and config 13 were often close, so the threshold is deliberately conservative.

The reason is architectural: the A6000 has 84 SMs and the MiniMax QKV has N/256 = 84 output tiles. The ordinary identity tile already provides good coverage; Stream-K adds reduction/scheduling overhead without solving an occupancy problem.

I also built a local sm86-only CUTLASS wheel with this selector branch:

if (m >= 12288 && n >= 16384 && k >= 4096 && k <= 8192 && current_device_is_sm86()) {
    return 0;
}

Validation of the rebuilt wheel:

QKV M=80661: 144.3 ms -> 89.2 ms
Attention out: ~33.5 ms -> ~33.6 ms
MLP up:       ~118.0 ms -> ~117.6 ms
MLP down:     ~66.8 ms -> ~67.0 ms

The selector branch is scoped to sm86 and leaves other architectures unchanged. The source also needs explicit sm86 in the default CUDA architecture list; the current Linux default omitted it and built 75,80,89,90a,100f,120f.

The local patch and complete CSV tuning matrix are available if maintainers want them.

@Many0therFunctions commented on GitHub (Aug 7, 2026): ## Additional sm86 tuning results I completed a resumable sweep on an RTX A6000 (sm86), covering: - `M = 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 49152, 65536, 80661, 98304` - all four MiniMax H3 projection geometries - all 14 compiled CUTLASS configurations - 3 randomized passes per configuration The relevant configurations are: - config 0: `128x256x64`, ordinary identity scheduling - config 13: `128x256x64`, Lean Stream-K scheduling The stable sm86 result is: ```text M >= 12288, N >= 16384, 4096 <= K <= 8192 => config 0 is consistently preferable to config 13 ``` I then validated neighboring geometries: ```text N: 16384, 20480, 21248, 21504, 21760, 24576, 28672 K: 4096, 5120, 5376, 6144, 7168, 8192 M: 8192, 12288, 16384, 32768 ``` For `M >= 12288`, config 0 beat config 13 by approximately 1.62x-1.76x across the tested region. At `M=8192`, config 0 and config 13 were often close, so the threshold is deliberately conservative. The reason is architectural: the A6000 has 84 SMs and the MiniMax QKV has `N/256 = 84` output tiles. The ordinary identity tile already provides good coverage; Stream-K adds reduction/scheduling overhead without solving an occupancy problem. I also built a local `sm86`-only CUTLASS wheel with this selector branch: ```cpp if (m >= 12288 && n >= 16384 && k >= 4096 && k <= 8192 && current_device_is_sm86()) { return 0; } ``` Validation of the rebuilt wheel: ```text QKV M=80661: 144.3 ms -> 89.2 ms Attention out: ~33.5 ms -> ~33.6 ms MLP up: ~118.0 ms -> ~117.6 ms MLP down: ~66.8 ms -> ~67.0 ms ``` The selector branch is scoped to sm86 and leaves other architectures unchanged. The source also needs explicit `sm86` in the default CUDA architecture list; the current Linux default omitted it and built `75,80,89,90a,100f,120f`. The local patch and complete CSV tuning matrix are available if maintainers want them.

bsakaguchi referenced this issue

2026-10-26 22:20:15 +00:00

[PR #30] Fix __version__ to read from package metadata instead of hardcoded string #90