mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #949] [Bug] RuntimeError: CUDA error: invalid device function #674
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#674
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Z27-DSR1 on GitHub (Mar 21, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/949
Originally assigned to: @Atream on GitHub.
检查清单
问题描述
运行 python ./ktransformers/local_chat.py --model_path /home/zdsr1/DeepSeek-R1 --gguf_path /home/zdsr1/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S --cpu_infer 56 --max_new_tokens 512 --force_think true 成功加载完成后，在Chat：对话过程中，出现如下错误，
Chat: 你好，你是谁？
Traceback (most recent call last):
File "/home/zdsr1/ktransformers/./ktransformers/local_chat.py", line 185, in
fire.Fire(local_chat)
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/local_chat.py", line 179, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/util/utils.py", line 208, in prefill_and_generate
logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/util/utils.py", line 165, in chunk_prefill
logits = model(
^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/operators/models.py", line 729, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/operators/attention.py", line 599, in forward
return self.forward_windows(
^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/operators/attention.py", line 539, in forward_windows
return self.forward_chunck(
^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/operators/attention.py", line 90, in forward_chunck
q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/anaconda3/envs/kt/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zdsr1/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 105, in forward
hidden_states = hidden_states.to(torch.float32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.请问如何解决这个问题。
复现步骤
python ./ktransformers/local_chat.py --model_path /home/zdsr1/DeepSeek-R1 --gguf_path /home/zdsr1/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S --cpu_infer 56 --max_new_tokens 512 --force_think true
环境信息
Win10 WSL2 Ubuntu22.04
双路Xeon Platinu 8260/767G内存/Nvidia Quadro RTX5000 16G显存
CUDA12.4/Pytorch2.5/Python311
@Atream commented on GitHub (Apr 29, 2025):
这是只有较旧的GPU会遇到的bug，目前没有统一的解决方案，之前有人通过先运行GPU版llama.cpp解决部分环境问题跑通了，但不确定是否通用。
目前我们没有ampere之前的GPU，所以暂时无法调试。
This is a bug that only older GPUs encounter. There is currently no universal solution. Previously, some people managed to resolve part of the environment issues and get it running by first executing the GPU version of
llama.cpp, but it is uncertain whether this method is universally applicable.At present, we do not have access to GPUs older than the Ampere architecture, so we are temporarily unable to debug it.