[GH-ISSUE #131] Flash Attention Decode availability detection raises ValueError for CPU devices instead of returning a boolean #41

New issue

Closed

opened 2026-10-26 22:19:35 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 22:19:35 +00:00

Owner

Originally created by @chaObserv on GitHub (Aug 23, 2026).
Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/131

Issue

The issue occurs in the following function:
github.com/Comfy-Org/comfy-kitchen@7d86acf60c/comfy_kitchen/flash_attention.py (L17)

When the device argument is CPU and torch is built with cuda 13.0 or later, the following error is raised at the return statement:
ValueError: Expected a cuda device, but got: cpu

Example

When using the ACE-Step 1.5 workflow template in ComfyUI, executing the TextEncodeAceStepAudio1.5 node results in the error.
This occurs because the text encoders are running on the CPU.

[INFO] pytorch version: 2.13.0+cu130
[INFO] Device: cuda:0 NVIDIA GeForce GTX 1660 Ti : cudaMallocAsync

[INFO] got prompt
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
[INFO] Requested to load ACE15TEModel_
[INFO] loaded completely;  4673.04 MB loaded, full load: True
[INFO] CLIP/text encoder model load device: cpu, offload device: cpu, current: cpu, dtype: torch.float16
[ERROR] !!! Exception during processing !!! Expected a cuda device, but got: cpu
[ERROR] Traceback (most recent call last):
  File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 545, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 344, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 318, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 306, in process_inputs
    result = f(**inputs)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy_api\internal\__init__.py", line 149, in wrapped_func
    return method(locked_class, **inputs)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy_api\latest\_io.py", line 1990, in EXECUTE_NORMALIZED
    to_return = cls.execute(*args, **kwargs)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy_extras\nodes_ace.py", line 60, in execute
    conditioning = clip.encode_from_tokens_scheduled(tokens)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 341, in encode_from_tokens_scheduled
    pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 410, in encode_from_tokens
    o = self.cond_stage_model.encode_token_weights(tokens)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\ace15.py", line 335, in encode_token_weights
    audio_codes = generate_audio_codes(getattr(self, self.lm_model, self.qwen3_06b), token_weight_pairs["lm_prompt"], token_weight_pairs["lm_prompt_negative"], min_tokens=lm_metadata["min_tokens"], max_tokens=lm_metadata["min_tokens"], seed=lm_metadata["seed"], cfg_scale=lm_metadata["cfg_scale"], temperature=lm_metadata["temperature"], top_p=lm_metadata["top_p"], top_k=lm_metadata["top_k"], min_p=lm_metadata["min_p"])
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\ace15.py", line 165, in generate_audio_codes
    return sample_manual_loop_no_classes(model, ids, cfg_scale=cfg_scale, temperature=temperature, top_p=top_p, top_k=top_k, min_p=min_p, seed=seed, min_tokens=min_tokens, max_new_tokens=max_tokens)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\ace15.py", line 64, in sample_manual_loop_no_classes
    past_key_values = model.transformer.model.init_kv_cache(embeds_batch, embeds.shape[1] + max_new_tokens, device, execution_dtype)
  File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\llama.py", line 802, in init_kv_cache
    fixed_kv = self.fixed_kv and comfy_kitchen.flash_attention_decode_is_available(device)
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\comfy_kitchen\flash_attention.py", line 22, in is_available
    return torch.cuda.get_device_capability(device) >= _MINIMUM_CAPABILITY
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\__init__.py", line 720, in get_device_capability
    prop = get_device_properties(device)
  File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\__init__.py", line 738, in get_device_properties
    device = _get_device_index(device, optional=True)
  File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\_utils.py", line 582, in _get_device_index
    raise ValueError(f"Expected a cuda device, but got: {device}")
ValueError: Expected a cuda device, but got: cpu

Originally created by @chaObserv on GitHub (Aug 23, 2026). Original GitHub issue: https://github.com/Comfy-Org/comfy-kitchen/issues/131 ### Issue The issue occurs in the following function: https://github.com/Comfy-Org/comfy-kitchen/blob/7d86acf60c88fd6c3c733c0e54db22ef74b8d77f/comfy_kitchen/flash_attention.py#L17 When the `device` argument is CPU and torch is built with cuda 13.0 or later, the following error is raised at the return statement: ```ValueError: Expected a cuda device, but got: cpu``` ### Example When using the ACE-Step 1.5 workflow template in ComfyUI, executing the `TextEncodeAceStepAudio1.5` node results in the error. This occurs because the text encoders are running on the CPU. - Log ```python [INFO] pytorch version: 2.13.0+cu130 [INFO] Device: cuda:0 NVIDIA GeForce GTX 1660 Ti : cudaMallocAsync [INFO] got prompt [INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float32 [INFO] Requested to load ACE15TEModel_ [INFO] loaded completely; 4673.04 MB loaded, full load: True [INFO] CLIP/text encoder model load device: cpu, offload device: cpu, current: cpu, dtype: torch.float16 [ERROR] !!! Exception during processing !!! Expected a cuda device, but got: cpu [ERROR] Traceback (most recent call last): File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 545, in execute output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 344, in get_output_data return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 318, in _async_map_node_over_list await process_inputs(input_dict, i) File "Path\ComfyUI_windows_portable\ComfyUI\execution.py", line 306, in process_inputs result = f(**inputs) File "Path\ComfyUI_windows_portable\ComfyUI\comfy_api\internal\__init__.py", line 149, in wrapped_func return method(locked_class, **inputs) File "Path\ComfyUI_windows_portable\ComfyUI\comfy_api\latest\_io.py", line 1990, in EXECUTE_NORMALIZED to_return = cls.execute(*args, **kwargs) File "Path\ComfyUI_windows_portable\ComfyUI\comfy_extras\nodes_ace.py", line 60, in execute conditioning = clip.encode_from_tokens_scheduled(tokens) File "Path\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 341, in encode_from_tokens_scheduled pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True) File "Path\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 410, in encode_from_tokens o = self.cond_stage_model.encode_token_weights(tokens) File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\ace15.py", line 335, in encode_token_weights audio_codes = generate_audio_codes(getattr(self, self.lm_model, self.qwen3_06b), token_weight_pairs["lm_prompt"], token_weight_pairs["lm_prompt_negative"], min_tokens=lm_metadata["min_tokens"], max_tokens=lm_metadata["min_tokens"], seed=lm_metadata["seed"], cfg_scale=lm_metadata["cfg_scale"], temperature=lm_metadata["temperature"], top_p=lm_metadata["top_p"], top_k=lm_metadata["top_k"], min_p=lm_metadata["min_p"]) File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\ace15.py", line 165, in generate_audio_codes return sample_manual_loop_no_classes(model, ids, cfg_scale=cfg_scale, temperature=temperature, top_p=top_p, top_k=top_k, min_p=min_p, seed=seed, min_tokens=min_tokens, max_new_tokens=max_tokens) File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\ace15.py", line 64, in sample_manual_loop_no_classes past_key_values = model.transformer.model.init_kv_cache(embeds_batch, embeds.shape[1] + max_new_tokens, device, execution_dtype) File "Path\ComfyUI_windows_portable\ComfyUI\comfy\text_encoders\llama.py", line 802, in init_kv_cache fixed_kv = self.fixed_kv and comfy_kitchen.flash_attention_decode_is_available(device) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\comfy_kitchen\flash_attention.py", line 22, in is_available return torch.cuda.get_device_capability(device) >= _MINIMUM_CAPABILITY ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\__init__.py", line 720, in get_device_capability prop = get_device_properties(device) File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\__init__.py", line 738, in get_device_properties device = _get_device_index(device, optional=True) File "Path\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\_utils.py", line 582, in _get_device_index raise ValueError(f"Expected a cuda device, but got: {device}") ValueError: Expected a cuda device, but got: cpu ```

bsakaguchi closed this issue

2026-10-26 22:19:36 +00:00

bsakaguchi commented

2026-10-26 22:19:36 +00:00

Author

Owner

@chaObserv commented on GitHub (Sep 26, 2026):

f61028a7b0 resolves it.

@chaObserv commented on GitHub (Sep 26, 2026): f61028a7b0f4be4beb3595c64ee919e0c345f96d resolves it.

bsakaguchi referenced this issue

2026-10-26 22:20:22 +00:00

[PR #41] [MERGED] Nibble Swizzle #98