mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #801] [Bug] RuntimeError: CUDA error: invalid device function #580
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#580
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @kaixwangwei on GitHub (Mar 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/801
Originally assigned to: @Atream on GitHub.
检查清单
问题描述
运行环境：
GPU:NVIDIA GeForce GTX 1050 Ti
CPU:12th Gen Intel(R) Core(TM) i5-12500
RAM：32GB
运行命令：
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ~/aCodes/huggleface/DeepSeek-V2-Lite-Q4_K_M-GGUF/
出现如下报错，请帮忙看看是啥原因？
Chat: who are you . for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 184, in <module> fire.Fire(local_chat) File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 178, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 208, in prefill_and_generate logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 165, in chunk_prefill logits = model( ^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 727, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 594, in forward return self.forward_windows( ^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 533, in forward_windows return self.forward_chunck( ^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 92, in forward_chunck compressed_kv = self.kv_a_proj_with_mqa(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 507, in forward y = self.generate_linear.forward(x) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangwei/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 327, in forward padding_input[:,:self.orin_in_features] = x ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: invalid device function CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile withTORCH_USE_CUDA_DSAto enable device-side assertions.复现步骤
环境步骤：https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/install.md
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ~/aCodes/huggleface/DeepSeek-V2-Lite-Q4_K_M-GGUF/
环境信息
运行环境：
GPU:NVIDIA GeForce GTX 1050 Ti
CPU:12th Gen Intel(R) Core(TM) i5-12500
@loverara commented on GitHub (Mar 5, 2025):
一样的问题
@kosehy commented on GitHub (Mar 8, 2025):
When I use
Quadro P5000, I also face same issue@kosehy commented on GitHub (Mar 8, 2025):
I followed #374 and Pyjacc's guideline in #425 and finally run the demo code using P5000 pascal architecture.
@save-FGG commented on GitHub (Mar 12, 2025):
我也遇到了，换了模型，改了运行参数还是一问问题就提示这个