mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #875] [Bug] CUDA error: too many blocks in cooperative launch(72) when use multi GPU #631
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#631
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Zzzer0o on GitHub (Mar 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/875
Originally assigned to: @Atream on GitHub.
Checklist
Describe the bug
Call Stack:
2025-03-13 02:14:27,620 - INFO - flashinfer.jit: Loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False
2025-03-13 02:14:27,717 - INFO - flashinfer.jit: Finished loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False
CUDA Error: too many blocks in cooperative launch (720) /workspace/flashinfer/flashinfer/data/include/flashinfer/attention/mla.cuh: line 1034 at function cudaLaunchCooperativeKernel((void*)kernel, nblks, nthrs, args, smem_size, stream)
Traceback (most recent call last):
File "/workspace/ktransformers/ktransformers/util/cuda_graph_runner.py", line 41, in capture
logits=model(inputs_embeds=inputs_embeds,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
outputs = self.model(
^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/operators/models.py", line 727, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/operators/attention.py", line 606, in forward
return self.forward_linux_flashinfer(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/operators/attention.py", line 437, in forward_linux_flashinfer
attn_output = self.mla_wrapper.run(q_nope, q_pe, compressed_kv, k_pe).view(bsz, q_len, self.num_heads, self.kv_lora_rank)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/operators/flashinfer_wrapper.py", line 144, in run
return self.wrapper.run(q_nope, q_pe, ckv, k_pe, return_lse = return_lse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/flashinfer/flashinfer/mla.py", line 352, in run
self._cached_module.run(
File "/opt/conda/lib/python3.11/site-packages/torch/_ops.py", line 1116, in call
return self._op(*args, **(kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/utils/_device.py", line 106, in torch_function
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/_ops.py", line 1116, in call
return self._op(*args, **(kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to run MLA, error: too many blocks in cooperative launch
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspace/ktransformers/ktransformers/local_chat.py", line 184, in
fire.Fire(local_chat)
File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/local_chat.py", line 173, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/ktransformers/ktransformers/util/utils.py", line 247, in prefill_and_generate
cuda_graph_runner.capture(model, next_token.unsqueeze(0), position_ids, cache_position, past_key_values, torch_device, return_dict=False, use_cache=True)
File "/workspace/ktransformers/ktransformers/util/cuda_graph_runner.py", line 40, in capture
with torch.cuda.graph(self.graph, stream = capture_stream):
File "/opt/conda/lib/python3.11/site-packages/torch/cuda/graphs.py", line 186, in exit
self.cuda_graph.capture_end()
File "/opt/conda/lib/python3.11/site-packages/torch/cuda/graphs.py", line 84, in capture_end
super().capture_end()
RuntimeError: CUDA error: operation failed due to a previous error during capture
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.docker version:
approachingai/ktransformers:v0.2.3post1-AVX512
Command:
python local_chat.py --optimize_path "./optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml" --cpu_infer 34 --max_new_tokens 1000 --force_think true (model path and gguf path omit)
Reproduction
model: DeepSeek-R1-Q4_K_M
Environment
cuda version: 12.1
GPU list:
GeForce RTX 3090 VRAM 24G
GeForce RTX 4080 VRAM 16G
TORCH_CUDA_ARCH_LIST=8.6 8.9
@Atream commented on GitHub (Mar 15, 2025):
Thank you for reporting the issue. We will reproduce and fix it as soon as possible.