mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #208] RuntimeError: CUDA error: invalid device function For first message #122
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#122
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Xxianna on GitHub (Feb 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/208
Environment
Error Log
just after first message. I got this error both with driver-cuda 12.4 and 12.8, and model unsloth q2k and q4k.
Chat: 你好
Traceback (most recent call last):
File "/home/u/prj/ktransformers/./ktransformers/local_chat.py", line 280, in
fire.Fire(local_chat)
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/local_chat.py", line 274, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/util/utils.py", line 150, in prefill_and_generate
logits = model(
^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/operators/models.py", line 722, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/operators/attention.py", line 372, in forward
return self.forward_chunck(
^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/operators/attention.py", line 273, in forward_chunck
q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/miniconda3/envs/deepseekfp8/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/u/prj/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 105, in forward
hidden_states = hidden_states.to(torch.float32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
@murongweibo commented on GitHub (Feb 13, 2025):
我也遇到这个错误了，是不是对gpu硬件版本有要求？我是在colab申请了一个T4跑的
报错信息：
根据readme文件快速开始跑的结果，挑的DeepSeek-V2-Lite-Chat.Q4_K_M.gguf模型（原来的地址模型是不存在的，估计是大佬已经删了，这里没更新，我换了一个）
@Xxianna commented on GitHub (Feb 13, 2025):
另外我把local_chat.py中设置bf16为默认格式的地方都设为float16了。但是这似乎没有什么帮助。
.to(torch.float32)报错行看起来很奇怪，只是一个转换，单独在torch里执行是没问题的
不知道是否和KLinearMarlin有关？我似乎在另一个issue里看到相关的建议
@Xxianna commented on GitHub (Feb 13, 2025):
看起来有相关问题
以及看到有人说是flash attention 2 不支持低版本的设备
@20000419 commented on GitHub (Feb 15, 2025):
请问您这个问题现在解决了么
@Titianzi commented on GitHub (Feb 15, 2025):
V100，也是相同报错，估计就是不支持flash_attn导致的
@Xxianna commented on GitHub (Feb 16, 2025):
我跑deepseek v2lite也报错，flash_attn有没有问题不知道，ktransformer里那个linear类的.cu感觉得修改才能支持
直接替换lineartorch计算全nan了，这两个类应该不一样吧
虽然准备下单4090了但还是期待一波支持20系
@motu001 commented on GitHub (Mar 26, 2025):
有人解决了吗