kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #441] running errors 运行错误 #304

New issue

Closed

opened 2026-10-29 14:37:37 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:37:37 +00:00

Owner

Originally created by @xialang2012 on GitHub (Feb 18, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/441

Win10 + 4090 + 512 GB内存，支持AMX，四通道
CUDA 12.4 + PyTorch 2.6

使用 https://modelscope.cn/models/unsloth/DeepSeek-R1-GGUF/files 中的DeepSeek-R1-Q4_K_M模型;
所有gguf文件均校验过hash256;
模型文件来自 https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main 里面所有的非tensor文件都下载了。

编译安装，pip list
Package Version

accelerate 1.3.0
aiohappyeyeballs 2.4.6
aiohttp 3.11.12
aiosignal 1.3.2
annotated-types 0.7.0
ansicon 1.89.0
anyio 4.8.0
attrs 25.1.0
blessed 1.20.0
build 1.2.2.post1
certifi 2025.1.31
charset-normalizer 3.4.1
click 8.1.8
cmake 3.31.4
colorama 0.4.6
colorlog 6.9.0
cpufeature 0.2.1
einops 0.8.1
fastapi 0.115.8
filelock 3.17.0
fire 0.7.0
flash_attn 2.7.4.post1
frozenlist 1.5.0
fsspec 2025.2.0
greenlet 3.1.1
h11 0.14.0
httpcore 1.0.7
httpx 0.28.1
huggingface-hub 0.28.1
idna 3.10
Jinja2 3.1.5
jinxed 1.3.0
jsonpatch 1.33
jsonpointer 3.0.0
ktransformers 0.2.0+cu124torch26fancy
langchain 0.3.18
langchain-core 0.3.35
langchain-text-splitters 0.3.6
langsmith 0.3.8
MarkupSafe 3.0.2
mpmath 1.3.0
multidict 6.1.0
networkx 3.4.2
ninja 1.11.1.3
numpy 1.26.4
orjson 3.10.15
packaging 24.2
pillow 11.0.0
pip 25.0
propcache 0.2.1
protobuf 5.29.3
psutil 7.0.0
pybind11 2.13.6
pydantic 2.10.6
pydantic_core 2.27.2
pyproject_hooks 1.2.0
PyYAML 6.0.2
regex 2024.11.6
requests 2.32.3
requests-toolbelt 1.0.0
safetensors 0.5.2
sentencepiece 0.2.0
setuptools 75.8.0
six 1.17.0
sniffio 1.3.1
SQLAlchemy 2.0.38
starlette 0.45.3
sympy 1.13.1
tenacity 9.0.0
termcolor 2.5.0
tokenizers 0.19.1
torch 2.6.0+cu124
torchaudio 2.6.0+cu124
torchvision 0.21.0+cu124
tqdm 4.67.1
transformers 4.43.2
triton 3.2.0
typing_extensions 4.12.2
urllib3 2.3.0
uvicorn 0.34.0
wcwidth 0.2.13
wheel 0.45.1
yarl 1.18.3
zstandard 0.23.0

运行命令后显示Chat，此时内存占用才不到50GB，输出 hello后就崩了。
C:\actions-runner_work\pytorch\pytorch\pytorch\aten\src\ATen\native\cuda\TensorCompare.cu:110: block: [0,0,0], thread: [0,0,0] Assertion input[0] != 0 failed.
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "C:\Users\admin\ktransformers\ktransformers\local_chat.py", line 182, in
fire.Fire(local_chat)
File "C:\Users\admin\miniconda3\envs\ktransformers\Lib\site-packages\fire\core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\miniconda3\envs\ktransformers\Lib\site-packages\fire\core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\miniconda3\envs\ktransformers\Lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\ktransformers\ktransformers\local_chat.py", line 176, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\ktransformers\ktransformers\util\utils.py", line 172, in prefill_and_generate
next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Originally created by @xialang2012 on GitHub (Feb 18, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/441 Win10 + 4090 + 512 GB内存，支持AMX，四通道 CUDA 12.4 + PyTorch 2.6 使用 https://modelscope.cn/models/unsloth/DeepSeek-R1-GGUF/files 中的DeepSeek-R1-Q4_K_M模型; 所有gguf文件均校验过hash256; 模型文件来自 https://huggingface.co/deepseek-ai/DeepSeek-R1/tree/main 里面所有的非tensor文件都下载了。编译安装，pip list Package Version ------------------------ ----------------------- accelerate 1.3.0 aiohappyeyeballs 2.4.6 aiohttp 3.11.12 aiosignal 1.3.2 annotated-types 0.7.0 ansicon 1.89.0 anyio 4.8.0 attrs 25.1.0 blessed 1.20.0 build 1.2.2.post1 certifi 2025.1.31 charset-normalizer 3.4.1 click 8.1.8 cmake 3.31.4 colorama 0.4.6 colorlog 6.9.0 cpufeature 0.2.1 einops 0.8.1 fastapi 0.115.8 filelock 3.17.0 fire 0.7.0 flash_attn 2.7.4.post1 frozenlist 1.5.0 fsspec 2025.2.0 greenlet 3.1.1 h11 0.14.0 httpcore 1.0.7 httpx 0.28.1 huggingface-hub 0.28.1 idna 3.10 Jinja2 3.1.5 jinxed 1.3.0 jsonpatch 1.33 jsonpointer 3.0.0 ktransformers 0.2.0+cu124torch26fancy langchain 0.3.18 langchain-core 0.3.35 langchain-text-splitters 0.3.6 langsmith 0.3.8 MarkupSafe 3.0.2 mpmath 1.3.0 multidict 6.1.0 networkx 3.4.2 ninja 1.11.1.3 numpy 1.26.4 orjson 3.10.15 packaging 24.2 pillow 11.0.0 pip 25.0 propcache 0.2.1 protobuf 5.29.3 psutil 7.0.0 pybind11 2.13.6 pydantic 2.10.6 pydantic_core 2.27.2 pyproject_hooks 1.2.0 PyYAML 6.0.2 regex 2024.11.6 requests 2.32.3 requests-toolbelt 1.0.0 safetensors 0.5.2 sentencepiece 0.2.0 setuptools 75.8.0 six 1.17.0 sniffio 1.3.1 SQLAlchemy 2.0.38 starlette 0.45.3 sympy 1.13.1 tenacity 9.0.0 termcolor 2.5.0 tokenizers 0.19.1 torch 2.6.0+cu124 torchaudio 2.6.0+cu124 torchvision 0.21.0+cu124 tqdm 4.67.1 transformers 4.43.2 triton 3.2.0 typing_extensions 4.12.2 urllib3 2.3.0 uvicorn 0.34.0 wcwidth 0.2.13 wheel 0.45.1 yarl 1.18.3 zstandard 0.23.0 运行命令后显示Chat，此时内存占用才不到50GB，输出 hello后就崩了。 C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\cuda\TensorCompare.cu:110: block: [0,0,0], thread: [0,0,0] Assertion `input[0] != 0` failed. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "C:\Users\admin\ktransformers\ktransformers\local_chat.py", line 182, in <module> fire.Fire(local_chat) File "C:\Users\admin\miniconda3\envs\ktransformers\Lib\site-packages\fire\core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\admin\miniconda3\envs\ktransformers\Lib\site-packages\fire\core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\admin\miniconda3\envs\ktransformers\Lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\admin\ktransformers\ktransformers\local_chat.py", line 176, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\admin\ktransformers\ktransformers\util\utils.py", line 172, in prefill_and_generate next_token = torch.multinomial(probs, num_samples=1).squeeze(1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

bsakaguchi

2026-10-29 14:37:37 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:37:38 +00:00

Author

Owner

@DechinPhy commented on GitHub (Feb 18, 2025):

Same Issue

@DechinPhy commented on GitHub (Feb 18, 2025): Same Issue

bsakaguchi commented

2026-10-29 14:37:38 +00:00

Author

Owner

@jprorikon commented on GitHub (Feb 18, 2025):

Same Issue on https://huggingface.co/failspy/mixtral-8x22b-v0.1-instruct-oh-GGUF

@jprorikon commented on GitHub (Feb 18, 2025): Same Issue on https://huggingface.co/failspy/mixtral-8x22b-v0.1-instruct-oh-GGUF

bsakaguchi commented

2026-10-29 14:37:38 +00:00

Author

Owner

@konghr commented on GitHub (Feb 20, 2025):

same issue on DeepSeek-R1-Q8_0

@konghr commented on GitHub (Feb 20, 2025): same issue on DeepSeek-R1-Q8_0

bsakaguchi commented

2026-10-29 14:37:38 +00:00

Author

Owner

@sunnsi commented on GitHub (Feb 22, 2025):

Same issue on DeepSeek-R1-Q2_K_L when multi-GPU is used.

/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [0,0,0], thread: [31,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
ERROR:    Exception in ASGI application

Works fine when single-GPU is used

@sunnsi commented on GitHub (Feb 22, 2025): Same issue on DeepSeek-R1-Q2_K_L when multi-GPU is used. ```bash /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [0,0,0], thread: [31,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed. ERROR: Exception in ASGI application ``` Works fine when single-GPU is used

bsakaguchi commented

2026-10-29 14:37:38 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！