[GH-ISSUE #782] [Bug] Newly added IQ1-2 quants produces garbage output using Unsloth quants and default optimize rules #565

New issue

Closed

opened 2026-10-29 14:41:16 +00:00 by bsakaguchi · 10 comments

bsakaguchi commented 2026-10-29 14:41:16 +00:00

Owner

Copy link

Originally created by @ThomasBaruzier on GitHub (Mar 4, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/782

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
• 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

I wanted to try https://github.com/kvcache-ai/ktransformers/pull/754

However, the local chat demo produces garbage outputs when using /optimize/optimize_rules/DeepSeek-V3-Chat.yaml with the following command:

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/files/ai/llama/models/ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512

Under /home/user/files/ai/llama/models/ there is DeepSeek-R1-UD-IQ2_XXS.gguf.

The same bug applies for DeepSeek-R1-UD-IQ1_S.gguf, and everything works when using DeepSeek-R1-UD-Q2_K_XL.gguf and the exact same setup.

Also, the DRAM usage is off. Loading DeepSeek-R1-UD-IQ1_S.gguf uses 14GB DRAM and DeepSeek-R1-UD-IQ2_XXS.gguf uses 88GB DRAM, which is definitely suspicious. Could it be that /optimize/optimize_rules/DeepSeek-V3-Chat.yaml needs modifications to work properly with such quants?

Example output:

Chat: hi
<think>
2025-03-04 01:11:41,690 - INFO - flashinfer.jit: Loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64
/home/user/files/ai/ktransformers/.venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/home/user/files/ai/ktransformers/.venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-03-04 01:12:04,751 - INFO - flashinfer.jit: Finished loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64
ggggggggggggggggggggggggggggggGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGPLESGGPLESGGPLESGGPLESGG999wise000ethelessethelessethelessighb000ethelessmaalethelessethelessourableomaticfoisethelessethelessmaalmaalmaalimatewiseethelessmaalmaalocatedmaalmaalmaalmaalmaalmaalmaalmaalormaismaalationallyationallyourablemaalmaalmaalmaalitedmaalfoismaalmaalaticallymaalocatedmaalourablefficialmaalationallyationallyviamentemaalighbmaalationallyaticallymaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalIVES000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000|---|---emsetWikimediaPLESIUMGGmium999000GGmium000000GGGG000000GGmium000GGGG000000GGGGMMMM000GGGGGGGG GGGGGG000000GGGGGGGG000000GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGMMMMPLES GG GGGG GGGGGG GGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GG GG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGÃGGÃGGGGGG GGGGGG GGGG �Ãí à à à ÃÃÃÃà à à � à à à à à �à à â � à à à ÃÃà à à à à à à Ãà ÃÃà ÃÃà à à à Ãà à à à à à à à à Ãà à à à à à Ãà à à ÃÃà Ãà ÃÃÃÃà à Ãà à Ãà ÃÃÃÃÃà ÃÃÃÃÃà à ÃÃà à ÃÃÃÃÃÃÃÃà à à ÃÃÃÃÃÃÃà Ãà Ãà à à à ÃÃÃÃà Ãà Ãà ÃÃÃÃÃà ÃÃà ÃÃÃÃÃÃÃÃÃÃÃà ÃÃà ÃÃÃÃà ÃÃÃÃÃÃÃÃà à Ãà à à à ÃÃà ÃÃà ÃÃÃà à ÃÃÃà à à ÃÃÃÃà à à 
prompt eval count:    8 token(s)
prompt eval duration: 0.7072505950927734s
prompt eval rate:     11.31140794438017 tokens/s
eval count:           512 token(s)
eval duration:        66.80018138885498s
eval rate:            7.664649846076955 tokens/s

Using these models in llama.cpp do not present any issues.

Reproduction

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/files/ai/llama/models/ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512

Environment

ArchLinux x86_64
Cuda 12.8
Driver 570.124.04
Python 3.12.9
Torch 2.6.0
KTransformers @ file:///home/user/files/ai/ktransformers

3 x RTX 3090
128GB DDR4
Ryzen 9 5950x

Originally created by @ThomasBaruzier on GitHub (Mar 4, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/782 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug I wanted to try https://github.com/kvcache-ai/ktransformers/pull/754 However, the local chat demo produces garbage outputs when using `/optimize/optimize_rules/DeepSeek-V3-Chat.yaml` with the following command: ``` python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/files/ai/llama/models/ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512 ``` Under `/home/user/files/ai/llama/models/` there is `DeepSeek-R1-UD-IQ2_XXS.gguf`. The same bug applies for `DeepSeek-R1-UD-IQ1_S.gguf`, and everything works when using `DeepSeek-R1-UD-Q2_K_XL.gguf` and the exact same setup. Also, the DRAM usage is off. Loading `DeepSeek-R1-UD-IQ1_S.gguf` uses 14GB DRAM and `DeepSeek-R1-UD-IQ2_XXS.gguf` uses 88GB DRAM, which is definitely suspicious. Could it be that `/optimize/optimize_rules/DeepSeek-V3-Chat.yaml` needs modifications to work properly with such quants? Example output: ``` Chat: hi <think> 2025-03-04 01:11:41,690 - INFO - flashinfer.jit: Loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64 /home/user/files/ai/ktransformers/.venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( /home/user/files/ai/ktransformers/.venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( 2025-03-04 01:12:04,751 - INFO - flashinfer.jit: Finished loading JIT ops: batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64 ggggggggggggggggggggggggggggggGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGPLESGGPLESGGPLESGGPLESGG999wise000ethelessethelessethelessighb000ethelessmaalethelessethelessourableomaticfoisethelessethelessmaalmaalmaalimatewiseethelessmaalmaalocatedmaalmaalmaalmaalmaalmaalmaalmaalormaismaalationallyationallyourablemaalmaalmaalmaalitedmaalfoismaalmaalaticallymaalocatedmaalourablefficialmaalationallyationallyviamentemaalighbmaalationallyaticallymaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalmaalIVES000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000|---|---emsetWikimediaPLESIUMGGmium999000GGmium000000GGGG000000GGmium000GGGG000000GGGGMMMM000GGGGGGGG GGGGGG000000GGGGGGGG000000GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGMMMMPLES GG GGGG GGGGGG GGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GG GG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGÃGGÃGGGGGG GGGGGG GGGG �Ãí à à à ÃÃÃÃà à à � à à à à à �à à â � à à à ÃÃà à à à à à à Ãà ÃÃà ÃÃà à à à Ãà à à à à à à à à Ãà à à à à à Ãà à à ÃÃà Ãà ÃÃÃÃà à Ãà à Ãà ÃÃÃÃÃà ÃÃÃÃÃà à ÃÃà à ÃÃÃÃÃÃÃÃà à à ÃÃÃÃÃÃÃà Ãà Ãà à à à ÃÃÃÃà Ãà Ãà ÃÃÃÃÃà ÃÃà ÃÃÃÃÃÃÃÃÃÃÃà ÃÃà ÃÃÃÃà ÃÃÃÃÃÃÃÃà à Ãà à à à ÃÃà ÃÃà ÃÃÃà à ÃÃÃà à à ÃÃÃÃà à à prompt eval count: 8 token(s) prompt eval duration: 0.7072505950927734s prompt eval rate: 11.31140794438017 tokens/s eval count: 512 token(s) eval duration: 66.80018138885498s eval rate: 7.664649846076955 tokens/s ``` Using these models in llama.cpp do not present any issues. ### Reproduction ``` python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/files/ai/llama/models/ --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512 ``` ### Environment ArchLinux x86_64 Cuda 12.8 Driver 570.124.04 Python 3.12.9 Torch 2.6.0 KTransformers @ file:///home/user/files/ai/ktransformers 3 x RTX 3090 128GB DDR4 Ryzen 9 5950x

bsakaguchi 2026-10-29 14:41:16 +00:00

• closed this issue
• added the
  stale
  label

bsakaguchi commented 2026-10-29 14:41:17 +00:00

Author

Owner

Copy link

@Azure-Tang commented on GitHub (Mar 4, 2025):

I'm testing UD-IQ1_S, it works fine here. Can you create a clean env to rebuild KT and try again?

<!-- gh-comment-id:2696070036 --> @Azure-Tang commented on GitHub (Mar 4, 2025): I'm testing UD-IQ1_S, it works fine here. Can you create a clean env to rebuild KT and try again?

bsakaguchi commented 2026-10-29 14:41:17 +00:00

Author

Owner

Copy link

@ThomasBaruzier commented on GitHub (Mar 4, 2025):

Thanks for your response, but I am still having issues with a clean install and this script:

#!/bin/bash

set -e

[ ! -d '.venv' ] && python3.11 -m venv .venv
source .venv/bin/activate

if [ ! -d 'repo' ]; then
  git clone 'https://github.com/kvcache-ai/ktransformers' repo
else
  git -C repo pull
fi

cd repo
git submodule init
git submodule update

pip install --upgrade pip wheel packaging ninja torch cpufeature numpy
pip install --no-build-isolation flash_attn
bash 'install.sh'

echo 'Done!'

Command for IQ1_S quant:

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/ --optimize_config_path repo/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512

Model:

~ du -bm /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf
133736  /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf

Chat: Hi
<think>
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGγγGGberryGGwayGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGiliaryγγGGGGGGGGGGGGGGGGGGGGPLES GGGGGGGGGGGGPLESGGGGGGPLESGG000000GGGGPLESGGGG000000GGGGGGGGPLESGGGGGGGGGGPLESGGGGGGGGGGGGPLESGGGGGGGGGGfficialGGGGGGPLESGGGGPLESGGGGGGGGPLESGGGGGGmaalGGfficial GGGGGG000GG000GGPLESGGGGPLESGGGGPLESGGGGwardsGGethelessetheless GGGGGGwardsGG000GGGGmaalGGfois000GGGGmaalGGfficialtechnologyGGmaalGGmaalGGmaal GGmaal GGmaal GGmaal GGmaalGGdomsemeemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemannemesmaalGGfficialtechnologyemesemesemannemes�emesmiumemespieces��emesethelessfois GW�mons�emesmaaletheless000GGmonsemesemesemesemesemesemesemesemesemesemesemesemesemesmanaemes�monsemesemset
prompt eval count:    9 token(s)
prompt eval duration: 0.42627763748168945s
prompt eval rate:     21.11300056265933 tokens/s
eval count:           512 token(s)
eval duration:        17.11542057991028s
eval rate:            29.914543882197954 tokens/s

<!-- gh-comment-id:2697234100 --> @ThomasBaruzier commented on GitHub (Mar 4, 2025): Thanks for your response, but I am still having issues with a clean install and this script: ```shell #!/bin/bash set -e [ ! -d '.venv' ] && python3.11 -m venv .venv source .venv/bin/activate if [ ! -d 'repo' ]; then git clone 'https://github.com/kvcache-ai/ktransformers' repo else git -C repo pull fi cd repo git submodule init git submodule update pip install --upgrade pip wheel packaging ninja torch cpufeature numpy pip install --no-build-isolation flash_attn bash 'install.sh' echo 'Done!' ``` Command for IQ1_S quant: ``` python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/ --optimize_config_path repo/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512 ``` Model: ``` ~ du -bm /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf 133736 /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf ``` ``` Chat: Hi <think> GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGγγGGberryGGwayGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGiliaryγγGGGGGGGGGGGGGGGGGGGGPLES GGGGGGGGGGGGPLESGGGGGGPLESGG000000GGGGPLESGGGG000000GGGGGGGGPLESGGGGGGGGGGPLESGGGGGGGGGGGGPLESGGGGGGGGGGfficialGGGGGGPLESGGGGPLESGGGGGGGGPLESGGGGGGmaalGGfficial GGGGGG000GG000GGPLESGGGGPLESGGGGPLESGGGGwardsGGethelessetheless GGGGGGwardsGG000GGGGmaalGGfois000GGGGmaalGGfficialtechnologyGGmaalGGmaalGGmaal GGmaal GGmaal GGmaal GGmaalGGdomsemeemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemannemesmaalGGfficialtechnologyemesemesemannemes�emesmiumemespieces��emesethelessfois GW�mons�emesmaaletheless000GGmonsemesemesemesemesemesemesemesemesemesemesemesemesemesmanaemes�monsemesemset prompt eval count: 9 token(s) prompt eval duration: 0.42627763748168945s prompt eval rate: 21.11300056265933 tokens/s eval count: 512 token(s) eval duration: 17.11542057991028s eval rate: 29.914543882197954 tokens/s ``` 

bsakaguchi commented 2026-10-29 14:41:17 +00:00

Author

Owner

Copy link

@Azure-Tang commented on GitHub (Mar 4, 2025):

Thanks for your response, but I am still having issues with a clean install and this script:

#!/bin/bash

set -e

[ ! -d '.venv' ] && python3.11 -m venv .venv
source .venv/bin/activate

if [ ! -d 'repo' ]; then
git clone 'https://github.com/kvcache-ai/ktransformers' repo
else
git -C repo pull
fi

cd repo
git submodule init
git submodule update

pip install --upgrade pip wheel packaging ninja torch cpufeature numpy
pip install --no-build-isolation flash_attn
bash 'install.sh'

echo 'Done!'
Command for IQ1_S quant:

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/ --optimize_config_path repo/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512

Model:

~ du -bm /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf
133736  /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf

Chat: Hi
<think>
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGγγGGberryGGwayGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGiliaryγγGGGGGGGGGGGGGGGGGGGGPLES GGGGGGGGGGGGPLESGGGGGGPLESGG000000GGGGPLESGGGG000000GGGGGGGGPLESGGGGGGGGGGPLESGGGGGGGGGGGGPLESGGGGGGGGGGfficialGGGGGGPLESGGGGPLESGGGGGGGGPLESGGGGGGmaalGGfficial GGGGGG000GG000GGPLESGGGGPLESGGGGPLESGGGGwardsGGethelessetheless GGGGGGwardsGG000GGGGmaalGGfois000GGGGmaalGGfficialtechnologyGGmaalGGmaalGGmaal GGmaal GGmaal GGmaal GGmaalGGdomsemeemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemannemesmaalGGfficialtechnologyemesemesemannemes�emesmiumemespieces��emesethelessfois GW�mons�emesmaaletheless000GGmonsemesemesemesemesemesemesemesemesemesemesemesemesemesmanaemes�monsemesemset
prompt eval count:    9 token(s)
prompt eval duration: 0.42627763748168945s
prompt eval rate:     21.11300056265933 tokens/s
eval count:           512 token(s)
eval duration:        17.11542057991028s
eval rate:            29.914543882197954 tokens/s

Seems that you didn't run bash install.sh?

<!-- gh-comment-id:2697323460 --> @Azure-Tang commented on GitHub (Mar 4, 2025): > Thanks for your response, but I am still having issues with a clean install and this script: > > #!/bin/bash > > set -e > > [ ! -d '.venv' ] && python3.11 -m venv .venv > source .venv/bin/activate > > if [ ! -d 'repo' ]; then > git clone 'https://github.com/kvcache-ai/ktransformers' repo > else > git -C repo pull > fi > > cd repo > git submodule init > git submodule update > > pip install --upgrade pip wheel packaging ninja torch cpufeature numpy > pip install --no-build-isolation flash_attn > bash 'install.sh' > > echo 'Done!' > Command for IQ1_S quant: > > ``` > python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/ --optimize_config_path repo/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --force_think True --use_cuda_graph True --cpu_infer 24 --max_new_tokens 512 > ``` > > Model: > > ``` > ~ du -bm /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf > 133736 /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf > ``` > > ``` > Chat: Hi > <think> > GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGγγGGberryGGwayGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGiliaryγγGGGGGGGGGGGGGGGGGGGGPLES GGGGGGGGGGGGPLESGGGGGGPLESGG000000GGGGPLESGGGG000000GGGGGGGGPLESGGGGGGGGGGPLESGGGGGGGGGGGGPLESGGGGGGGGGGfficialGGGGGGPLESGGGGPLESGGGGGGGGPLESGGGGGGmaalGGfficial GGGGGG000GG000GGPLESGGGGPLESGGGGPLESGGGGwardsGGethelessetheless GGGGGGwardsGG000GGGGmaalGGfois000GGGGmaalGGfficialtechnologyGGmaalGGmaalGGmaal GGmaal GGmaal GGmaal GGmaalGGdomsemeemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemesemannemesmaalGGfficialtechnologyemesemesemannemes�emesmiumemespieces��emesethelessfois GW�mons�emesmaaletheless000GGmonsemesemesemesemesemesemesemesemesemesemesemesemesemesmanaemes�monsemesemset > prompt eval count: 9 token(s) > prompt eval duration: 0.42627763748168945s > prompt eval rate: 21.11300056265933 tokens/s > eval count: 512 token(s) > eval duration: 17.11542057991028s > eval rate: 29.914543882197954 tokens/s > ``` > >  Seems that you didn't run `bash install.sh`?

bsakaguchi commented 2026-10-29 14:41:17 +00:00

Author

Owner

Copy link

@ThomasBaruzier commented on GitHub (Mar 4, 2025):

@Azure-Tang I did, it's in the script I provided. It says that the KTransformer package has been built and installed in my python environment

Testing llama.cpp with the same model produces coherent outputs:

./llama.cpp/llama-cli \
  -m /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf \
  -t 16 --prio 2 --temp 0.7 -b 31 -ub 31 --no-mmap \
  -c 4096 -ctk q4_0 -ngl 24 -mg 0 -ts 8,8,8

> Hi
<think>

</think>

Hello! How can I assist you today?

<!-- gh-comment-id:2697339734 --> @ThomasBaruzier commented on GitHub (Mar 4, 2025): @Azure-Tang I did, it's in the script I provided. It says that the KTransformer package has been built and installed in my python environment Testing llama.cpp with the same model produces coherent outputs: ``` ./llama.cpp/llama-cli \ -m /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S.gguf \ -t 16 --prio 2 --temp 0.7 -b 31 -ub 31 --no-mmap \ -c 4096 -ctk q4_0 -ngl 24 -mg 0 -ts 8,8,8 ``` ``` > Hi <think> </think> Hello! How can I assist you today? ```

bsakaguchi commented 2026-10-29 14:41:17 +00:00

Author

Owner

Copy link

@Azure-Tang commented on GitHub (Mar 4, 2025):

Use conda instead or try this:

#!/bin/bash

set -e

[ ! -d '.venv' ] && python3.11 -m venv .venv
source .venv/bin/activate

if [ ! -d 'repo' ]; then
  git clone 'https://github.com/kvcache-ai/ktransformers' repo
else
  git -C repo pull
fi

cd repo
git submodule init
git submodule update

pip install --upgrade pip wheel packaging ninja torch cpufeature numpy
pip install --no-build-isolation flash_attn
which python # add this one
which pip
bash 'install.sh'

echo 'Done!'

I suspect that your env is not switching.

<!-- gh-comment-id:2697415775 --> @Azure-Tang commented on GitHub (Mar 4, 2025): Use conda instead or try this: ```shell #!/bin/bash set -e [ ! -d '.venv' ] && python3.11 -m venv .venv source .venv/bin/activate if [ ! -d 'repo' ]; then git clone 'https://github.com/kvcache-ai/ktransformers' repo else git -C repo pull fi cd repo git submodule init git submodule update pip install --upgrade pip wheel packaging ninja torch cpufeature numpy pip install --no-build-isolation flash_attn which python # add this one which pip bash 'install.sh' echo 'Done!' ``` I suspect that your env is not switching.

bsakaguchi commented 2026-10-29 14:41:17 +00:00

Author

Owner

Copy link

@ThomasBaruzier commented on GitHub (Mar 5, 2025):

I confirm the venv is used, Arch does not accept global envs and KTransformers is not installed on this scope.

I used Docker to try again:

build.sh:

#!/bin/bash

archs=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader)
[ -z "$archs" ] && echo "No GPU detected" && exit 1
archs="${archs//$'\n'/ }"

if [ "$1" = 'rebuild' ]; then
  if [ -z "$STY" ]; then
    screen -L -Logfile dockerbuild.log \
      -mS ktransformers-build bash "$0" "$@"
    exit 0
  fi

  docker compose down
  rm -rf repo

  docker compose build --build-arg "ARCHS=$archs"
  if [ "$?" != 0 ]; then read; exit 1; fi
else
  docker compose down
fi

docker compose up -d

compose.yml.venv:

services:
  ktransformers:
    build:
      dockerfile: Dockerfile
      args:
        USER: "$USER"
    volumes:
      - ./DeepSeek-R1-UD-IQ1_S.gguf:/models/DeepSeek-R1-UD-IQ1_S.gguf
      - ./repo:/home/ktransformers
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

Dockerfile.venv:

FROM archlinux:latest

ARG USER=ktransformers
ARG ARCHS=changeme

RUN yes | pacman -Syu base-devel cmake git sudo cuda \
    && useradd -m -G wheel ${USER} \
    && echo "${USER} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

USER ${USER}
WORKDIR /home/${USER}

ENV MAKEFLAGS -j$(nproc)
ENV CUDA_HOME /opt/cuda
ENV TORCH_CUDA_ARCH_LIST=${ARCHS}

RUN git clone https://aur.archlinux.org/yay-bin.git \
    && cd yay-bin \
    && yes | makepkg -si \
    && cd .. \
    && rm -rf yay-bin

RUN yes | yay -S python311 \
    && yes | yay -Scc

RUN python3.11 -m venv venv

RUN source venv/bin/activate && \
    pip install wheel packaging ninja torch cpufeature numpy

RUN source venv/bin/activate && \
    pip install --no-build-isolation flash_attn

RUN source venv/bin/activate && \
    git clone 'https://github.com/kvcache-ai/ktransformers' repo && \
    cd repo && \
    git submodule init && \
    git submodule update && \
    bash install.sh

CMD ["tail", "-f"]

Then, docker exec -it docker-ktransformers-1 bash, source venv/bin/activate, and finally the same command as before with IQ1_S. Sadly, same garbage output.

I was unable to use anaconda or miniconda due to unkown errors and missing symbols such as the following: https://github.com/pytorch/pytorch/issues/123097

<!-- gh-comment-id:2699583275 --> @ThomasBaruzier commented on GitHub (Mar 5, 2025): I confirm the venv is used, Arch does not accept global envs and KTransformers is not installed on this scope. I used Docker to try again: `build.sh`: ```sh #!/bin/bash archs=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader) [ -z "$archs" ] && echo "No GPU detected" && exit 1 archs="${archs//$'\n'/ }" if [ "$1" = 'rebuild' ]; then if [ -z "$STY" ]; then screen -L -Logfile dockerbuild.log \ -mS ktransformers-build bash "$0" "$@" exit 0 fi docker compose down rm -rf repo docker compose build --build-arg "ARCHS=$archs" if [ "$?" != 0 ]; then read; exit 1; fi else docker compose down fi docker compose up -d ``` `compose.yml.venv`: ```venv services: ktransformers: build: dockerfile: Dockerfile args: USER: "$USER" volumes: - ./DeepSeek-R1-UD-IQ1_S.gguf:/models/DeepSeek-R1-UD-IQ1_S.gguf - ./repo:/home/ktransformers deploy: resources: reservations: devices: - capabilities: [gpu] ``` `Dockerfile.venv`: ```venv FROM archlinux:latest ARG USER=ktransformers ARG ARCHS=changeme RUN yes | pacman -Syu base-devel cmake git sudo cuda \ && useradd -m -G wheel ${USER} \ && echo "${USER} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers USER ${USER} WORKDIR /home/${USER} ENV MAKEFLAGS -j$(nproc) ENV CUDA_HOME /opt/cuda ENV TORCH_CUDA_ARCH_LIST=${ARCHS} RUN git clone https://aur.archlinux.org/yay-bin.git \ && cd yay-bin \ && yes | makepkg -si \ && cd .. \ && rm -rf yay-bin RUN yes | yay -S python311 \ && yes | yay -Scc RUN python3.11 -m venv venv RUN source venv/bin/activate && \ pip install wheel packaging ninja torch cpufeature numpy RUN source venv/bin/activate && \ pip install --no-build-isolation flash_attn RUN source venv/bin/activate && \ git clone 'https://github.com/kvcache-ai/ktransformers' repo && \ cd repo && \ git submodule init && \ git submodule update && \ bash install.sh CMD ["tail", "-f"] ``` Then, `docker exec -it docker-ktransformers-1 bash`, `source venv/bin/activate`, and finally the same command as before with IQ1_S. Sadly, same garbage output. I was unable to use anaconda or miniconda due to unkown errors and missing symbols such as the following: https://github.com/pytorch/pytorch/issues/123097

bsakaguchi commented 2026-10-29 14:41:18 +00:00

Author

Owner

Copy link

@Azure-Tang commented on GitHub (Mar 5, 2025):

Try use latest image and rebuild in container?
docker pull approachingai/ktransformers

dont use venv in container.

<!-- gh-comment-id:2699682725 --> @Azure-Tang commented on GitHub (Mar 5, 2025): Try use latest image and rebuild in container? `docker pull approachingai/ktransformers` dont use venv in container.

bsakaguchi commented 2026-10-29 14:41:18 +00:00

Author

Owner

Copy link

@ThomasBaruzier commented on GitHub (Mar 5, 2025):

I did a clean build from scratch using miniconda3 and the ktransformers wiki on arch, no containers:

Env:

(ktransformers) ~/files/ai/ktransformers pip freeze
Using Python 3.11.11 environment at: /home/user/.conda/envs/ktransformers
accelerate==1.4.0
annotated-types==0.7.0
anyio==4.8.0
blessed==1.20.0
blobfile==3.0.0
build==1.2.2.post1
certifi==2025.1.31
charset-normalizer==3.4.1
click==8.1.8
colorlog==6.9.0
cpufeature==0.2.1
einops==0.8.1
fastapi==0.115.11
filelock==3.13.1
fire==0.7.0
flash-attn==2.7.4.post1
fsspec==2024.6.1
greenlet==3.1.1
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
huggingface-hub==0.29.1
idna==3.10
jinja2==3.1.4
jsonpatch==1.33
jsonpointer==3.0.0
ktransformers @ file:///home/user/files/ai/ktransformers/repo
langchain==0.3.20
langchain-core==0.3.41
langchain-text-splitters==0.3.6
langsmith==0.3.11
lxml==5.3.1
markupsafe==2.1.5
mpmath==1.3.0
networkx==3.3
ninja==1.11.1.3
numpy==2.1.2
nvidia-cublas-cu12==12.6.4.1
nvidia-cuda-cupti-cu12==12.6.80
nvidia-cuda-nvrtc-cu12==12.6.77
nvidia-cuda-runtime-cu12==12.6.77
nvidia-cudnn-cu12==9.5.1.17
nvidia-cufft-cu12==11.3.0.4
nvidia-curand-cu12==10.3.7.77
nvidia-cusolver-cu12==11.7.1.2
nvidia-cusparse-cu12==12.5.4.2
nvidia-cusparselt-cu12==0.6.3
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.6.85
nvidia-nvtx-cu12==12.6.77
orjson==3.10.15
packaging==24.2
pillow==11.0.0
pip @ file:///croot/pip_1737991979965/work
protobuf==6.30.0
psutil==7.0.0
pycryptodomex==3.21.0
pydantic==2.10.6
pydantic-core==2.27.2
pyproject-hooks==1.2.0
pyyaml==6.0.2
regex==2024.11.6
requests==2.32.3
requests-toolbelt==1.0.0
safetensors==0.5.3
sentencepiece==0.2.0
setuptools==75.8.0
six==1.17.0
sniffio==1.3.1
sqlalchemy==2.0.38
starlette==0.46.0
sympy==1.13.1
tenacity==9.0.0
termcolor==2.5.0
tiktoken==0.9.0
tokenizers==0.19.1
torch==2.6.0+cu126
torchaudio==2.6.0+cu126
torchvision==0.21.0+cu126
tqdm==4.67.1
transformers==4.43.2
triton==3.2.0
typing-extensions==4.12.2
urllib3==2.3.0
uvicorn==0.34.0
wcwidth==0.2.13
wheel==0.45.1
zstandard==0.23.0
(ktransformers) ~/files/ai/ktransformers which pip
/home/user/.conda/envs/ktransformers/bin/pip
(ktransformers) ~/files/ai/ktransformers which pip3
/home/user/.conda/envs/ktransformers/bin/pip3
(ktransformers) ~/files/ai/ktransformers which python
/home/user/.conda/envs/ktransformers/bin/python
(ktransformers) ~/files/ai/ktransformers which python3
/home/user/.conda/envs/ktransformers/bin/python3

Command:

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S-parts --use_cuda_graph False

Output:

Chat: Hi

prompt eval count:    6 token(s)
prompt eval duration: 0.406200647354126s
prompt eval rate:     14.771025204125774 tokens/s
eval count:           300 token(s)
eval duration:        35.0155770778656s
eval rate:            8.567615473904013 tokens/s
Chat: hi hi

prompt eval count:    7 token(s)
prompt eval duration: 0.13895964622497559s
prompt eval rate:     50.37433665214579 tokens/s
eval count:           300 token(s)
eval duration:        34.741849184036255s
eval rate:            8.635118942887152 tokens/s
Chat: Solve 3x+10=0              
/pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 184, in <module>
    fire.Fire(local_chat)
  File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 178, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 251, in prefill_and_generate
    tokens.append(int(next_token))
                  ^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

What's weird is that the RAM usage is still 10GB, even without optimization rules.
Also, the CUDA error here cannot be fixed by --use_cuda_graph False.

Once again, the exact same setup works for Q2_K_XL dynamic quant.

I will try the approachingai/ktransformers container and report back.

<!-- gh-comment-id:2700385034 --> @ThomasBaruzier commented on GitHub (Mar 5, 2025): I did a clean build from scratch using miniconda3 and the ktransformers wiki on arch, no containers: Env: ``` (ktransformers) ~/files/ai/ktransformers pip freeze Using Python 3.11.11 environment at: /home/user/.conda/envs/ktransformers accelerate==1.4.0 annotated-types==0.7.0 anyio==4.8.0 blessed==1.20.0 blobfile==3.0.0 build==1.2.2.post1 certifi==2025.1.31 charset-normalizer==3.4.1 click==8.1.8 colorlog==6.9.0 cpufeature==0.2.1 einops==0.8.1 fastapi==0.115.11 filelock==3.13.1 fire==0.7.0 flash-attn==2.7.4.post1 fsspec==2024.6.1 greenlet==3.1.1 h11==0.14.0 httpcore==1.0.7 httpx==0.28.1 huggingface-hub==0.29.1 idna==3.10 jinja2==3.1.4 jsonpatch==1.33 jsonpointer==3.0.0 ktransformers @ file:///home/user/files/ai/ktransformers/repo langchain==0.3.20 langchain-core==0.3.41 langchain-text-splitters==0.3.6 langsmith==0.3.11 lxml==5.3.1 markupsafe==2.1.5 mpmath==1.3.0 networkx==3.3 ninja==1.11.1.3 numpy==2.1.2 nvidia-cublas-cu12==12.6.4.1 nvidia-cuda-cupti-cu12==12.6.80 nvidia-cuda-nvrtc-cu12==12.6.77 nvidia-cuda-runtime-cu12==12.6.77 nvidia-cudnn-cu12==9.5.1.17 nvidia-cufft-cu12==11.3.0.4 nvidia-curand-cu12==10.3.7.77 nvidia-cusolver-cu12==11.7.1.2 nvidia-cusparse-cu12==12.5.4.2 nvidia-cusparselt-cu12==0.6.3 nvidia-nccl-cu12==2.21.5 nvidia-nvjitlink-cu12==12.6.85 nvidia-nvtx-cu12==12.6.77 orjson==3.10.15 packaging==24.2 pillow==11.0.0 pip @ file:///croot/pip_1737991979965/work protobuf==6.30.0 psutil==7.0.0 pycryptodomex==3.21.0 pydantic==2.10.6 pydantic-core==2.27.2 pyproject-hooks==1.2.0 pyyaml==6.0.2 regex==2024.11.6 requests==2.32.3 requests-toolbelt==1.0.0 safetensors==0.5.3 sentencepiece==0.2.0 setuptools==75.8.0 six==1.17.0 sniffio==1.3.1 sqlalchemy==2.0.38 starlette==0.46.0 sympy==1.13.1 tenacity==9.0.0 termcolor==2.5.0 tiktoken==0.9.0 tokenizers==0.19.1 torch==2.6.0+cu126 torchaudio==2.6.0+cu126 torchvision==0.21.0+cu126 tqdm==4.67.1 transformers==4.43.2 triton==3.2.0 typing-extensions==4.12.2 urllib3==2.3.0 uvicorn==0.34.0 wcwidth==0.2.13 wheel==0.45.1 zstandard==0.23.0 (ktransformers) ~/files/ai/ktransformers which pip /home/user/.conda/envs/ktransformers/bin/pip (ktransformers) ~/files/ai/ktransformers which pip3 /home/user/.conda/envs/ktransformers/bin/pip3 (ktransformers) ~/files/ai/ktransformers which python /home/user/.conda/envs/ktransformers/bin/python (ktransformers) ~/files/ai/ktransformers which python3 /home/user/.conda/envs/ktransformers/bin/python3 ``` Command: ``` python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/user/storage/models/gguf/DeepSeek-R1-UD-IQ1_S-parts --use_cuda_graph False ``` Output: ``` Chat: Hi prompt eval count: 6 token(s) prompt eval duration: 0.406200647354126s prompt eval rate: 14.771025204125774 tokens/s eval count: 300 token(s) eval duration: 35.0155770778656s eval rate: 8.567615473904013 tokens/s Chat: hi hi prompt eval count: 7 token(s) prompt eval duration: 0.13895964622497559s prompt eval rate: 50.37433665214579 tokens/s eval count: 300 token(s) eval duration: 34.741849184036255s eval rate: 8.635118942887152 tokens/s Chat: Solve 3x+10=0 /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 184, in <module> fire.Fire(local_chat) File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 178, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/home/user/.conda/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 251, in prefill_and_generate tokens.append(int(next_token)) ^^^^^^^^^^^^^^^ RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` What's weird is that the RAM usage is still 10GB, even without optimization rules. Also, the CUDA error here cannot be fixed by `--use_cuda_graph False`. Once again, the exact same setup works for Q2_K_XL dynamic quant. I will try the approachingai/ktransformers container and report back.

bsakaguchi commented 2026-10-29 14:41:18 +00:00

Author

Owner

Copy link

@ThomasBaruzier commented on GitHub (Mar 5, 2025):

Previous comment, TLDR I crashed because I used AVX512 instead of AVX2

Here are my results using approachingai/ktransformer:latest-AVX512 644b3538931c:

root@7423fdb3b2e0:/workspace# python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-UD-IQ1_S-parts/ --use_cuda_graph False
flashinfer not found, use triton for linux
tokenizer_config.json: 100%|���������������| 3.59k/3.59k [00:00<00:00, 35.1MB/s]
tokenizer.json: 100%|����������������������| 7.85M/7.85M [00:00<00:00, 16.7MB/s]
config.json: 100%|�������������������������| 1.73k/1.73k [00:00<00:00, 21.6MB/s]
configuration_deepseek.py: 100%|�����������| 10.6k/10.6k [00:00<00:00, 73.8MB/s]
A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-R1:
- configuration_deepseek.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Illegal instruction (core dumped)

root@7423fdb3b2e0:/workspace# python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-UD-Q2_K_XL/ --use_cuda_graph False
flashinfer not found, use triton for linux
using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Illegal instruction (core dumped)

root@fe4d60619c4f:/workspace# nvidia-smi
Wed Mar  5 10:44:42 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04             Driver Version: 570.124.04     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:01:00.0 Off |                  N/A |
| 50%   30C    P8             35W /  275W |       2MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:0A:00.0 Off |                  N/A |
| 40%   32C    P8             28W /  275W |       2MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        Off |   00000000:0B:00.0 Off |                  N/A |
| 30%   30C    P8             21W /  275W |       2MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

This time, all quants are crashing. I tried redownloading IQ_1S and not merging the splited files as well, without success.

Here are my results using approachingai/ktransformer:latest-AVX2 :

Chat: Solve 10x+3=0
Chat: Solve 10x+3=0
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/local_chat.py", line 183, in <module>
    fire.Fire(local_chat)
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/local_chat.py", line 177, in local_chat
    generated = prefill_and_generate(
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 234, in prefill_and_generate
    next_token = decode_one_tokens(cuda_graph_runner, next_token.unsqueeze(0), position_ids, cache_position, past_key_values, use_cuda_graph).to(torch_device)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 151, in decode_one_tokens
    next_token = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
root@f4045963d745:/workspace#

Edit: The process still uses around 15GB of RAM for unknown reasons

Edit: Even shorter prompts don't crash, but do not output anything:

Chat: Hi

prompt eval count:    6 token(s)
prompt eval duration: 0.3764495849609375s
prompt eval rate:     15.938389201896964 tokens/s
eval count:           300 token(s)
eval duration:        38.405927419662476s
eval rate:            7.811294249501982 tokens/s
Chat:

<!-- gh-comment-id:2700540844 --> @ThomasBaruzier commented on GitHub (Mar 5, 2025): <details> <summary> Previous comment, TLDR I crashed because I used AVX512 instead of AVX2</summary> Here are my results using `approachingai/ktransformer:latest-AVX512 644b3538931c`: ``` root@7423fdb3b2e0:/workspace# python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-UD-IQ1_S-parts/ --use_cuda_graph False flashinfer not found, use triton for linux tokenizer_config.json: 100%|���������������| 3.59k/3.59k [00:00<00:00, 35.1MB/s] tokenizer.json: 100%|����������������������| 7.85M/7.85M [00:00<00:00, 16.7MB/s] config.json: 100%|�������������������������| 1.73k/1.73k [00:00<00:00, 21.6MB/s] configuration_deepseek.py: 100%|�����������| 10.6k/10.6k [00:00<00:00, 73.8MB/s] A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-R1: - configuration_deepseek.py . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. using custom modeling_xxx.py. using default_optimize_rule for DeepseekV3ForCausalLM Injecting model as ktransformers.operators.models . KDeepseekV2Model Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Illegal instruction (core dumped) ``` ``` root@7423fdb3b2e0:/workspace# python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /models/DeepSeek-R1-UD-Q2_K_XL/ --use_cuda_graph False flashinfer not found, use triton for linux using custom modeling_xxx.py. using default_optimize_rule for DeepseekV3ForCausalLM Injecting model as ktransformers.operators.models . KDeepseekV2Model Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Illegal instruction (core dumped) ``` ``` root@fe4d60619c4f:/workspace# nvidia-smi Wed Mar 5 10:44:42 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.124.04 Driver Version: 570.124.04 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A | | 50% 30C P8 35W / 275W | 2MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 3090 Off | 00000000:0A:00.0 Off | N/A | | 40% 32C P8 28W / 275W | 2MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA GeForce RTX 3090 Off | 00000000:0B:00.0 Off | N/A | | 30% 30C P8 21W / 275W | 2MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ ``` This time, all quants are crashing. I tried redownloading IQ_1S and not merging the splited files as well, without success. </details> Here are my results using `approachingai/ktransformer:latest-AVX2 `: ``` Chat: Solve 10x+3=0 Chat: Solve 10x+3=0 Traceback (most recent call last): File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/opt/conda/lib/python3.10/site-packages/ktransformers/local_chat.py", line 183, in <module> fire.Fire(local_chat) File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/opt/conda/lib/python3.10/site-packages/ktransformers/local_chat.py", line 177, in local_chat generated = prefill_and_generate( File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 234, in prefill_and_generate next_token = decode_one_tokens(cuda_graph_runner, next_token.unsqueeze(0), position_ids, cache_position, past_key_values, use_cuda_graph).to(torch_device) File "/opt/conda/lib/python3.10/site-packages/ktransformers/util/utils.py", line 151, in decode_one_tokens next_token = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 root@f4045963d745:/workspace# ``` Edit: The process still uses around 15GB of RAM for unknown reasons Edit: Even shorter prompts don't crash, but do not output anything: ``` Chat: Hi prompt eval count: 6 token(s) prompt eval duration: 0.3764495849609375s prompt eval rate: 15.938389201896964 tokens/s eval count: 300 token(s) eval duration: 38.405927419662476s eval rate: 7.811294249501982 tokens/s Chat: ```

bsakaguchi commented 2026-10-29 14:41:18 +00:00

Author

Owner

Copy link

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

---

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

<!-- gh-comment-id:3639970458 --> @ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。 如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。 感谢您对 KTransformers 的贡献！

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#565

No description provided.