mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #971] Tesla P100一直跑不起来，能成功加载模型到出现chat的界面，但是一提交prompt就报错。torch,vision,audio是本地编译的，有指定架构，全部都单独测试过没问题。是ktransformers这边需要配… #688
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#688
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @2203476 on GitHub (Mar 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/971
@blime4 commented on GitHub (Mar 25, 2025):
You can provide the error log
@2203476 commented on GitHub (Mar 25, 2025):
Here is the error log:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/tg/GitProjects/ktransformers/ktransformers/local_chat.py", line 185, in
fire.Fire(local_chat)
File "/home/tg/miniconda3/envs/torchenv/lib/python3.11/site-packages/fire-0.7.0-py3.11.egg/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/miniconda3/envs/torchenv/lib/python3.11/site-packages/fire-0.7.0-py3.11.egg/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/tg/miniconda3/envs/torchenv/lib/python3.11/site-packages/fire-0.7.0-py3.11.egg/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/util/utils.py", line 208, in prefill_and_generate
logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/util/utils.py", line 165, in chunk_prefill
logits = model(
^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/operators/models.py", line 729, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/operators/attention.py", line 599, in forward
return self.forward_windows(
^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/operators/attention.py", line 539, in forward_windows
return self.forward_chunck(
^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/operators/attention.py", line 90, in forward_chunck
q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/pytorch-2.6.0/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tg/GitProjects/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 105, in forward
hidden_states = hidden_states.to(torch.float32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
(torchenv) tg@tg-Super-Server:$ echo $TORCH_USE_CUDA_DSA && echo $CUDA_LAUNCH_BLOCKING && echo $PYTORCH_USE_CUDA_DSA
1
1
1
(torchenv) tg@tg-Super-Server:$
@annian101 commented on GitHub (Mar 25, 2025):
解决了吗老哥？我也是这个报错
@swordfar commented on GitHub (Mar 25, 2025):
需要TF32精度支持。P100不支持，需要30系列（或者对标）显卡以上