mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#327
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @MtrsChJG on GitHub (Feb 18, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/465
Chat: 你能做什么？
您好！我是由中国的深度求索（DeepSeek）公司开发的智能助手DeepSeek-R1。有关 inherent capabilities, 我能够帮助您解答/pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion
probability tensor contains eitherinf,nanor element < 0failed.Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/inspur/lizezxun/LLM/ktransformers/ktransformers/local_chat.py", line 179, in
fire.Fire(local_chat)
File "/home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/inspur/lizezxun/LLM/ktransformers/ktransformers/local_chat.py", line 173, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/inspur/lizezxun/LLM/ktransformers/ktransformers/util/utils.py", line 199, in prefill_and_generate
next_token = decode_one_tokens(cuda_graph_runner, next_token.unsqueeze(0), position_ids, cache_position, past_key_values, use_cuda_graph).to(torch_device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/inspur/lizezxun/LLM/ktransformers/ktransformers/util/utils.py", line 126, in decode_one_tokens
next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x79d2e2caa1b6 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x79d2e2c53a76 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x79d2e2d98918 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x103ad78 (0x79d29003ad78 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x10433c5 (0x79d2900433c5 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #5: + 0x643582 (0x79d2d9a43582 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x6f30f (0x79d2e2c8b30f in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x79d2e2c8433b in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x79d2e2c844e9 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #9: + 0x904458 (0x79d2d9d04458 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x79d2d9d047a6 in /home/inspur/anaconda3/envs/ktransformers/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #11: python() [0x5c9364]
frame #12: python() [0x57de5a]
frame #13: python() [0x4f913a]
frame #14: python() [0x5fe2bc]
frame #15: python() [0x5fce76]
frame #19: + 0x2a1ca (0x79d2e3a2a1ca in /lib/x86_64-linux-gnu/libc.so.6)
frame #20: __libc_start_main + 0x8b (0x79d2e3a2a28b in /lib/x86_64-linux-gnu/libc.so.6)
frame #21: python() [0x5bbf93]
Aborted (core dumped)
@chuangzhidan commented on GitHub (Feb 19, 2025):
me2
@CmyLjj commented on GitHub (Feb 21, 2025):
@MtrsChJG hello, please tell me how to deal with this problem,i also have this problem
@tunantu commented on GitHub (Mar 14, 2025):
Hi, have you addressed this issue? I met the same problem.
@createthis commented on GitHub (Apr 22, 2025):
+1
I restarted
ktransformersand used the same exact prompt and it worked fine. Memory leak? It had been running for several days before this happened.