kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #148] Cuda Error #83

New issue

Closed

opened 2026-10-29 14:33:49 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:33:49 +00:00

Owner

Originally created by @stchip on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/148

I'm getting this error, using a 4090 windows

Build cuda_12.4.r12.4/compiler.33961263_0
torch 2.4.0+cu124
flash-attn 2.7.3

python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S" --cpu_infer 32 --cache_lens 1536

using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
..........
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
current target dtype: torch.bfloat16
loading token_embd.weight to cpu
E:\temp\ktransformers\ktransformers\util\custom_gguf.py:482: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:212.)
data = torch.from_numpy(data)
current target dtype: torch.bfloat16
......
loading blk.60.ffn_norm.weight to cuda:0
current target dtype: torch.bfloat16
loading output_norm.weight to cuda:0
current target dtype: torch.bfloat16
loading output.weight to cuda:0
Windows
Chat: What is 2+2?
Traceback (most recent call last):
File "E:\temp\ktransformers\ktransformers\local_chat.py", line 280, in
fire.Fire(local_chat)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "E:\temp\ktransformers\ktransformers\local_chat.py", line 274, in local_chat
generated = prefill_and_generate(
File "E:\temp\ktransformers\ktransformers\util\utils.py", line 160, in prefill_and_generate
logits = model(
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\models\modeling_deepseek_v3.py", line 1688, in forward
outputs = self.model(
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\operators\models.py", line 722, in forward
layer_outputs = decoder_layer(
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\models\modeling_deepseek_v3.py", line 1205, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\operators\attention.py", line 372, in forward
return self.forward_chunck(
File "E:\temp\ktransformers\ktransformers\operators\attention.py", line 273, in forward_chunck
q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "E:\temp\ktransformers\ktransformers\operators\linear.py", line 410, in forward
return self.generate_linear.forward(x)
File "E:\temp\ktransformers\ktransformers\operators\linear.py", line 227, in forward
x = KTransformersOps.gptq_marlin_gemm(
RuntimeError: gpt_marlin_gemm only supports bfloat16 and float16
Exception raised from gptq_marlin_gemm at E:\temp\ktransformers\ktransformers\ktransformers_ext\cuda\gptq_marlin\gptq_marlin.cu:1901 (most recent call first):
00007FF9CDA5462900007FF9CDA54580 c10.dll!c10::Error::Error [ @ ]
00007FF9CDA5416A00007FF9CDA54110 c10.dll!c10::detail::torchCheckFail [ @ ]
00007FF8048592AA00007FF8047FFC20 KTransformersOps.cp310-win_amd64.pyd!PyInit_KTransformersOps [ @ ]
00007FF804809B7E00007FF8047FFC20 KTransformersOps.cp310-win_amd64.pyd!PyInit_KTransformersOps [ @ ]
00007FF804809C1400007FF8047FFC20 KTransformersOps.cp310-win_amd64.pyd!PyInit_KTransformersOps [ @ ]
00007FF8047FD8DE00007FF8047F5680 KTransformersOps.cp310-win_amd64.pyd!c10::ivalue::Object::operator= [ @ ]
00007FF9BAAFC24A00007FF9BAAFC178 python310.dll!PyObject_IsTrue [ @ ]
00007FF9BAADF95E00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAADC8C500007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [ @ ]
00007FF9BAAD97C300007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB093B400007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [ @ ]
00007FF9BAAD97C300007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB093B400007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [ @ ]
00007FF9BAA8BA1700007FF9BAA8B9AC python310.dll!PyObject_FastCallDictTstate [ @ ]
00007FF9BABAD5C400007FF9BABAD548 python310.dll!PyObject_Call_Prepend [ @ ]
00007FF9BABAD52000007FF9BABAD04C python310.dll!PyBytesWriter_Resize [ @ ]
00007FF9BAADF95E00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [ @ ]
00007FF9BAAD97C300007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB093B400007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [ @ ]
00007FF9BAA8BA7F00007FF9BAA8B9AC python310.dll!PyObject_FastCallDictTstate [ @ ]
00007FF9BABAD5C400007FF9BABAD548 python310.dll!PyObject_Call_Prepend [ @ ]
00007FF9BABAD52000007FF9BABAD04C python310.dll!PyBytesWriter_Resize [ @ ]
00007FF9BAB0930700007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [ @ ]
00007FF9BAA8BA7F00007FF9BAA8B9AC python310.dll!PyObject_FastCallDictTstate [ @ ]
00007FF9BABAD5C400007FF9BABAD548 python310.dll!PyObject_Call_Prepend [ @ ]
00007FF9BABAD52000007FF9BABAD04C python310.dll!PyBytesWriter_Resize [ @ ]
00007FF9BAAC60DB00007FF9BAAC5F78 python310.dll!PyObject_MakeTpCall [ @ ]
00007FF9BAAE24FC00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]
00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [ @ ]
00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [ @ ]
00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [ @ ]
00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [ @ ]

Originally created by @stchip on GitHub (Feb 11, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/148 I'm getting this error, using a 4090 windows Build cuda_12.4.r12.4/compiler.33961263_0 torch 2.4.0+cu124 flash-attn 2.7.3 python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-IQ1_S" --cpu_infer 32 --cache_lens 1536 using custom modeling_xxx.py. using default_optimize_rule for DeepseekV3ForCausalLM Injecting model as ktransformers.operators.models . KDeepseekV2Model Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default .......... Injecting model.layers.60.post_attention_layernorm as default Injecting model.norm as default Injecting lm_head as default current target dtype: torch.bfloat16 loading token_embd.weight to cpu E:\temp\ktransformers\ktransformers\util\custom_gguf.py:482: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:212.) data = torch.from_numpy(data) current target dtype: torch.bfloat16 ...... loading blk.60.ffn_norm.weight to cuda:0 current target dtype: torch.bfloat16 loading output_norm.weight to cuda:0 current target dtype: torch.bfloat16 loading output.weight to cuda:0 Windows Chat: What is 2+2? Traceback (most recent call last): File "E:\temp\ktransformers\ktransformers\local_chat.py", line 280, in <module> fire.Fire(local_chat) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "E:\temp\ktransformers\ktransformers\local_chat.py", line 274, in local_chat generated = prefill_and_generate( File "E:\temp\ktransformers\ktransformers\util\utils.py", line 160, in prefill_and_generate logits = model( File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\models\modeling_deepseek_v3.py", line 1688, in forward outputs = self.model( File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\operators\models.py", line 722, in forward layer_outputs = decoder_layer( File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\models\modeling_deepseek_v3.py", line 1205, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\operators\attention.py", line 372, in forward return self.forward_chunck( File "E:\temp\ktransformers\ktransformers\operators\attention.py", line 273, in forward_chunck q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states))) File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "E:\temp\ktransformers\ktransformers\operators\linear.py", line 410, in forward return self.generate_linear.forward(x) File "E:\temp\ktransformers\ktransformers\operators\linear.py", line 227, in forward x = KTransformersOps.gptq_marlin_gemm( **RuntimeError: gpt_marlin_gemm only supports bfloat16 and float16 Exception raised from gptq_marlin_gemm at E:\temp\ktransformers\ktransformers\ktransformers_ext\cuda\gptq_marlin\gptq_marlin.cu:1901 (most recent call first):** 00007FF9CDA5462900007FF9CDA54580 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>] 00007FF9CDA5416A00007FF9CDA54110 c10.dll!c10::detail::torchCheckFail [<unknown file> @ <unknown line number>] 00007FF8048592AA00007FF8047FFC20 KTransformersOps.cp310-win_amd64.pyd!PyInit_KTransformersOps [<unknown file> @ <unknown line number>] 00007FF804809B7E00007FF8047FFC20 KTransformersOps.cp310-win_amd64.pyd!PyInit_KTransformersOps [<unknown file> @ <unknown line number>] 00007FF804809C1400007FF8047FFC20 KTransformersOps.cp310-win_amd64.pyd!PyInit_KTransformersOps [<unknown file> @ <unknown line number>] 00007FF8047FD8DE00007FF8047F5680 KTransformersOps.cp310-win_amd64.pyd!c10::ivalue::Object::operator= [<unknown file> @ <unknown line number>] 00007FF9BAAFC24A00007FF9BAAFC178 python310.dll!PyObject_IsTrue [<unknown file> @ <unknown line number>] 00007FF9BAADF95E00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAADC8C500007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>] 00007FF9BAAD97C300007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB093B400007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>] 00007FF9BAAD97C300007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB093B400007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>] 00007FF9BAA8BA1700007FF9BAA8B9AC python310.dll!PyObject_FastCallDictTstate [<unknown file> @ <unknown line number>] 00007FF9BABAD5C400007FF9BABAD548 python310.dll!PyObject_Call_Prepend [<unknown file> @ <unknown line number>] 00007FF9BABAD52000007FF9BABAD04C python310.dll!PyBytesWriter_Resize [<unknown file> @ <unknown line number>] 00007FF9BAADF95E00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>] 00007FF9BAAD97C300007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB093B400007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>] 00007FF9BAA8BA7F00007FF9BAA8B9AC python310.dll!PyObject_FastCallDictTstate [<unknown file> @ <unknown line number>] 00007FF9BABAD5C400007FF9BABAD548 python310.dll!PyObject_Call_Prepend [<unknown file> @ <unknown line number>] 00007FF9BABAD52000007FF9BABAD04C python310.dll!PyBytesWriter_Resize [<unknown file> @ <unknown line number>] 00007FF9BAB0930700007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAADAA9700007FF9BAADAA10 python310.dll!PyFunction_Vectorcall [<unknown file> @ <unknown line number>] 00007FF9BAA8BA7F00007FF9BAA8B9AC python310.dll!PyObject_FastCallDictTstate [<unknown file> @ <unknown line number>] 00007FF9BABAD5C400007FF9BABAD548 python310.dll!PyObject_Call_Prepend [<unknown file> @ <unknown line number>] 00007FF9BABAD52000007FF9BABAD04C python310.dll!PyBytesWriter_Resize [<unknown file> @ <unknown line number>] 00007FF9BAAC60DB00007FF9BAAC5F78 python310.dll!PyObject_MakeTpCall [<unknown file> @ <unknown line number>] 00007FF9BAAE24FC00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>] 00007FF9BAAE122D00007FF9BAADB960 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>] 00007FF9BAAD96E500007FF9BAAD8690 python310.dll!PyObject_GC_Malloc [<unknown file> @ <unknown line number>] 00007FF9BAB0941000007FF9BAB09358 python310.dll!PyVectorcall_Call [<unknown file> @ <unknown line number>] 00007FF9BAB0928B00007FF9BAB09148 python310.dll!PyObject_Call [<unknown file> @ <unknown line number>]

bsakaguchi

2026-10-29 14:33:49 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:33:50 +00:00

Author

Owner

@KMSorSMS commented on GitHub (Feb 12, 2025):

RuntimeError: gpt_marlin_gemm only supports bfloat16 and float16

We now support qx-k-m and IQ4. The other IQ_x is not supported yet. But we will support it in the following day.

you can find Q4 gguf from there https://huggingface.co/unsloth/DeepSeek-R1-GGUF

@KMSorSMS commented on GitHub (Feb 12, 2025): > RuntimeError: gpt_marlin_gemm only supports bfloat16 and float16 We now support qx-k-m and IQ4. The other IQ_x is not supported yet. But we will support it in the following day. <br> you can find Q4 gguf from there https://huggingface.co/unsloth/DeepSeek-R1-GGUF

bsakaguchi commented

2026-10-29 14:33:51 +00:00

Author

Owner

@stchip commented on GitHub (Feb 12, 2025):

Thanks for your reply.

I updated and used this
python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL" --cpu_infer 32 --cache_lens 1536

I got this error

Chat: hi
Traceback (most recent call last):
File "E:\temp\ktransformers\ktransformers\local_chat.py", line 179, in
fire.Fire(local_chat)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "E:\temp\ktransformers\ktransformers\local_chat.py", line 173, in local_chat
generated = prefill_and_generate(
File "E:\temp\ktransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate
next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either inf, nan or element < 0

@stchip commented on GitHub (Feb 12, 2025): Thanks for your reply. I updated and used this python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL" --cpu_infer 32 --cache_lens 1536 I got this error Chat: hi Traceback (most recent call last): File "E:\temp\ktransformers\ktransformers\local_chat.py", line 179, in <module> fire.Fire(local_chat) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "E:\temp\ktransformers\ktransformers\local_chat.py", line 173, in local_chat generated = prefill_and_generate( File "E:\temp\ktransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) **RuntimeError: probability tensor contains either `inf`, `nan` or element < 0**

bsakaguchi commented

2026-10-29 14:33:51 +00:00

Author

Owner

@ddht10086 commented on GitHub (Feb 13, 2025):

Thanks for your reply.

I updated and used this python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL" --cpu_infer 32 --cache_lens 1536

I got this error

Chat: hi Traceback (most recent call last): File "E:\temp\ktransformers\ktransformers\local_chat.py", line 179, in fire.Fire(local_chat) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "E:\temp\ktransformers\ktransformers\local_chat.py", line 173, in local_chat generated = prefill_and_generate( File "E:\temp\ktransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0

Same question

@ddht10086 commented on GitHub (Feb 13, 2025): > Thanks for your reply. > > I updated and used this python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL" --cpu_infer 32 --cache_lens 1536 > > I got this error > > Chat: hi Traceback (most recent call last): File "E:\temp\ktransformers\ktransformers\local_chat.py", line 179, in fire.Fire(local_chat) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "E:\temp\ktransformers\ktransformers\local_chat.py", line 173, in local_chat generated = prefill_and_generate( File "E:\temp\ktransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) **RuntimeError: probability tensor contains either `inf`, `nan` or element < 0** Same question

bsakaguchi commented

2026-10-29 14:33:51 +00:00

Author

Owner

@Pb-207 commented on GitHub (Feb 13, 2025):

Thanks for your reply.
I updated and used this python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL" --cpu_infer 32 --cache_lens 1536
I got this error
Chat: hi Traceback (most recent call last): File "E:\temp\ktransformers\ktransformers\local_chat.py", line 179, in fire.Fire(local_chat) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "E:\temp\ktransformers\ktransformers\local_chat.py", line 173, in local_chat generated = prefill_and_generate( File "E:\temp\ktransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0

Same question

Same issue:
python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path E:\LLM\models\DeepSeek\deepseek-r1-671B --port 2345 --cache_lens 1024 --cpu_infer 32
......
Chat:
Traceback (most recent call last):
File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\ktransformers\local_chat.py", line 179, in
fire.Fire(local_chat)
File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\venv\Lib\site-packages\fire\core.py", line 143, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\venv\Lib\site-packages\fire\core.py", line 477, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\venv\Lib\site-packages\fire\core.py", line 693, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\ktransformers\local_chat.py", line 173, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate
next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: probability tensor contains either inf, nan or element < 0

@Pb-207 commented on GitHub (Feb 13, 2025): > > Thanks for your reply. > > I updated and used this python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path "E:/temp/llama.cpp/DeepSeek-R1-GGUF/DeepSeek-R1-UD-Q2_K_XL" --cpu_infer 32 --cache_lens 1536 > > I got this error > > Chat: hi Traceback (most recent call last): File "E:\temp\ktransformers\ktransformers\local_chat.py", line 179, in fire.Fire(local_chat) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "E:\temp\ktransformers\ktransformers\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "E:\temp\ktransformers\ktransformers\local_chat.py", line 173, in local_chat generated = prefill_and_generate( File "E:\temp\ktransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) **RuntimeError: probability tensor contains either `inf`, `nan` or element < 0** > > Same question Same issue: python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path E:\LLM\models\DeepSeek\deepseek-r1-671B --port 2345 --cache_lens 1024 --cpu_infer 32 ...... Chat: Traceback (most recent call last): File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\ktransformers\local_chat.py", line 179, in <module> fire.Fire(local_chat) File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\venv\Lib\site-packages\fire\core.py", line 143, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\venv\Lib\site-packages\fire\core.py", line 477, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\venv\Lib\site-packages\fire\core.py", line 693, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\ktransformers\local_chat.py", line 173, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "D:\PyCharm Community Edition 2022.2.3\Projects\KTransformers\ktransformers\util\utils.py", line 168, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

bsakaguchi commented

2026-10-29 14:33:51 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！

bsakaguchi referenced this issue

2026-10-29 14:51:36 +00:00

[PR #83] [MERGED] Use cond var to avoid busy loop #1346