mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #475] RuntimeError: CUDA error: invalid device function.Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. #334
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#334
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @youde2000 on GitHub (Feb 18, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/475
加载参数之后，显示了Chat，但是输入任何内容按回车就会报错，
Chat:
Traceback (most recent call last):
File "/home/user/code/ktransformers/./ktransformers/local_chat.py", line 179, in
fire.Fire(local_chat)
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/user/code/ktransformers/./ktransformers/local_chat.py", line 173, in local_chat
generated = prefill_and_generate(
File "/home/user/code/ktransformers/./ktransformers/util/utils.py", line 154, in prefill_and_generate
logits = model(
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/code/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
outputs = self.model(
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/code/ktransformers/./ktransformers/operators/models.py", line 722, in forward
layer_outputs = decoder_layer(
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/code/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/code/ktransformers/./ktransformers/operators/attention.py", line 416, in forward
return self.forward_linux(
File "/home/user/code/ktransformers/./ktransformers/operators/attention.py", line 204, in forward_linux
q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/anaconda3/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/code/ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 105, in forward
hidden_states = hidden_states.to(torch.float32)
RuntimeError: CUDA error: invalid device function
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.@youde2000 commented on GitHub (Feb 18, 2025):
环境Ubuntu20.4+ 4090 + 512 GB内存
(py310) user@user-PowerEdge-R750:~$ pip list
Package Version Editable project location
accelerate 1.4.0
aiohappyeyeballs 2.4.6
aiohttp 3.11.12
aiosignal 1.3.2
annotated-types 0.7.0
anyio 4.8.0
async-timeout 4.0.3
attrs 25.1.0
blessed 1.20.0
Brotli 1.0.9
build 1.2.2.post1
certifi 2025.1.31
charset-normalizer 3.3.2
click 8.1.8
colorlog 6.9.0
cpufeature 0.2.1
einops 0.8.1
exceptiongroup 1.2.2
fastapi 0.115.8
filelock 3.13.1
fire 0.7.0
flash_attn 2.7.4.post1
frozenlist 1.5.0
fsspec 2025.2.0
gmpy2 2.2.1
greenlet 3.1.1
h11 0.14.0
httpcore 1.0.7
httpx 0.28.1
huggingface-hub 0.28.1
idna 3.7
Jinja2 3.1.5
jsonpatch 1.33
jsonpointer 3.0.0
ktransformers 0.2.0+cu124torch25fancy /home/user/code/ktransformers
langchain 0.3.19
langchain-core 0.3.35
langchain-text-splitters 0.3.6
langsmith 0.3.8
MarkupSafe 3.0.2
mkl_fft 1.3.11
mkl_random 1.2.8
mkl-service 2.4.0
mpmath 1.3.0
multidict 6.1.0
networkx 3.4.2
ninja 1.11.1.3
numpy 1.26.4
orjson 3.10.15
packaging 24.2
pillow 11.1.0
pip 25.0
propcache 0.2.1
protobuf 5.29.3
psutil 7.0.0
pydantic 2.10.6
pydantic_core 2.27.2
pyproject_hooks 1.2.0
PySocks 1.7.1
PyYAML 6.0.2
regex 2024.11.6
requests 2.32.3
requests-toolbelt 1.0.0
safetensors 0.5.2
sentencepiece 0.2.0
setuptools 75.8.0
six 1.17.0
sniffio 1.3.1
SQLAlchemy 2.0.38
starlette 0.45.3
sympy 1.13.1
tenacity 9.0.0
termcolor 2.5.0
tokenizers 0.19.1
tomli 2.2.1
torch 2.5.1
torchaudio 2.5.1
torchvision 0.20.1
tqdm 4.67.1
transformers 4.43.2
triton 3.1.0
typing_extensions 4.12.2
urllib3 2.3.0
uvicorn 0.34.0
wcwidth 0.2.13
wheel 0.45.1
yarl 1.18.3
zstandard 0.23.0
@songzibo commented on GitHub (Feb 19, 2025):
I have the same problem.
@111wang222 commented on GitHub (Feb 19, 2025):
俺也一样 V100s ，按照https://github.com/kvcache-ai/ktransformers/issues/374 这个部署了下，仍然报错
@wolfreka commented on GitHub (Feb 20, 2025):
Same thing bothered me. local chat shows "chat:", then show these things blow.
errors information:
.......
ktransformers/./ktransformers/models/modeling_deepseek_v3.py", line 105, in forward
hidden_states = hidden_states.to(torch.float32)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.@zwbsxwt commented on GitHub (Feb 28, 2025):
启动增加 这个参数试下 --use_cuda_graph=false
@loverara commented on GitHub (Mar 5, 2025):
有解决吗？一样的问题
@loverara commented on GitHub (Mar 5, 2025):
试过了不管用一样的报错
@zwbsxwt commented on GitHub (Mar 11, 2025):
我也遇到类似的问题了, 最后我通过计算sha256sum发现有个模型文件不完整重新下载就好了
@annian101 commented on GitHub (Mar 26, 2025):
我计算模型文件但是文件也没有问题