[GH-ISSUE #885] [Bug] 模型推理过程出现nan #642

New issue

Open

opened 2026-10-29 14:42:17 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:42:17 +00:00

Owner

Originally created by @tunantu on GitHub (Mar 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/885

检查清单

1. 我已经搜索过相关问题，但未能获得预期的帮助
2. 该问题在最新版本中尚未修复
3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

在部署DeepSeek-R1-Q4_K_M和DeepSeek-R1-Q8_0后，进行推理的时候报错：

File "/home/wangks/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/wangks/ktransformers/ktransformers/local_chat_bak.py", line 178, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "/home/wangks/ktransformers/ktransformers/util/utils.py", line 239, in prefill_and_generate
next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

/pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed.

同样的环境，运行：DeepSeek-V2-Lite-Chat.Q6_K_M不会出错。

我已经检查过，g++是符合版本要求的：
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

复现步骤

执行命令：python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4.yaml

模型：DeepSeek-R1-Q4_K_M和DeepSeek-R1-Q8_0

环境信息

g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.。

单张/4张4090。

CPU： Model name: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz

Originally created by @tunantu on GitHub (Mar 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/885 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述在部署DeepSeek-R1-Q4_K_M和DeepSeek-R1-Q8_0后，进行推理的时候报错： File "/home/wangks/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/wangks/ktransformers/ktransformers/local_chat_bak.py", line 178, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/home/wangks/ktransformers/ktransformers/util/utils.py", line 239, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed. 同样的环境，运行：DeepSeek-V2-Lite-Chat.Q6_K_M不会出错。我已经检查过，g++是符合版本要求的： g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ### 复现步骤执行命令：python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --optimize_config_path ./ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4.yaml 模型：DeepSeek-R1-Q4_K_M和DeepSeek-R1-Q8_0 ### 环境信息 g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.。单张/4张4090。 CPU： Model name: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz

bsakaguchi commented

2026-10-29 14:42:18 +00:00

Author

Owner

@Atream commented on GitHub (Mar 15, 2025):

请更新最新版本的代码并重新编译，为确保重新编译生效，建议先删除旧版本的编译结果。如果仍然有nan的问题，请提供prompt

@Atream commented on GitHub (Mar 15, 2025): 请更新最新版本的代码并重新编译，为确保重新编译生效，建议先删除旧版本的编译结果。如果仍然有nan的问题，请提供prompt

bsakaguchi commented

2026-10-29 14:42:18 +00:00

Author

Owner

@tunantu commented on GitHub (Mar 18, 2025):

问题已经解决。

原因：下载到本地的GGUF文件和huggingface上官方版本的文件相比，文件遭到了损害。

解决思路：通过sha256sum计算下载下来的每个gguf文件的sha56值，和官方文件公布的sha256值进行对比，定位出受损害的文件。

解决方法: 定位出受损害的文件后，重新下载该文件即可。

@tunantu commented on GitHub (Mar 18, 2025): 问题已经解决。原因：下载到本地的GGUF文件和huggingface上官方版本的文件相比，文件遭到了损害。解决思路：通过sha256sum计算下载下来的每个gguf文件的sha56值，和官方文件公布的sha256值进行对比，定位出受损害的文件。解决方法: 定位出受损害的文件后，重新下载该文件即可。