mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #649] 请问如何支持图灵架构的英伟达显卡2080ti？ #479
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#479
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @freepanzer on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/649
软件环境：
Ubuntu24.04 X86，CUDA12.4，conda安装python3.11的环境，显卡驱动版本：550.144.03，支持CUDA12.4,模型：Qwen2-57B-A14B-Instruct（该模型在ktransformers支持列表内，在optimize_rule的yaml可以看到）。
硬件环境：
Intel 10850K 3.6Ghz，128GB DDR4 3200内存，2TB固态Nvme,1TB固态Sata，显卡：NV2080TI 22GB.
上述软件环境可以正常运行Ollama 32B的deepseek模型。
我在B站看到有人在2080ti运行成功(https://www.bilibili.com/video/BV1ZKABePEK1/?spm_id_from=333.1391.0.0&vd_source=f7c588c075aa31c0f75ad170ea385d95)，
运行命令：python -m ktransformers.local_chat --model_name Qwen2-57B-A14B-Instruct_Q4 --model_path /media/freepanzer/ProgramTools/Model/Qwen2-57B-A14B-Instruct/hg_models --gguf_path /media/freepanzer/ProgramTools/Model/Qwen2-57B-A14B-Instruct/hg_models --optimize_rule_path /media/freepanzer/ProgramTools/Model/ktransformers-main/ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml，
但我的2080ti 22g从来没有成功过，可以预加载模型，运行到Chat，提问环节直接报错：
Chat: 你好
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 187, in
fire.Fire(local_chat)
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 181, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 156, in prefill_and_generate
logits = model(
^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_qwen2_moe.py", line 1414, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 362, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_qwen2_moe.py", line 921, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_qwen2_moe.py", line 426, in forward
query_states = self.q_proj(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 419, in forward
return self.generate_linear.forward(x)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/freepanzer/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/linear.py", line 250, in forward
x = x + self.bias
~~^~~~~~~~~~~
RuntimeError: CUDA error: invalid device function
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.@Azure-Tang commented on GitHub (Feb 25, 2025):
#150