[GH-ISSUE #693] gpu 利用率为0,不知道为什么？ #505

New issue

Closed

opened 2026-10-29 14:40:36 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented 2026-10-29 14:40:36 +00:00

Owner

Copy link

Originally created by @xiaohangguo on GitHub (Feb 26, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/693

您好，我按照官方给出的教程https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/install.md
配置了环境，并且正常跑起来了。但是显卡的利用率始终为0，感觉是有问题的。推理全靠cpu，token吐的速度很慢很慢。
加载完模型花费了一个小时。然后大概四五分钟推理出一个token。
运行时会输出一堆的：

“for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows”

详细见后面附件中的图
一些背景信息：
服务器：centos8
内存：256G
cpus core ：40
Model name: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
GPU：4090 48G

运行时信息：

环境版本如下：

lvshuhang@s04:~
$ conda list
# packages in environment at /home/lvshuhang/.conda/envs/ktransformers:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
accelerate                1.4.0                    pypi_0    pypi
aiohappyeyeballs          2.4.6                    pypi_0    pypi
aiohttp                   3.11.13                  pypi_0    pypi
aiosignal                 1.3.2                    pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
anyio                     4.8.0                    pypi_0    pypi
attrs                     25.1.0                   pypi_0    pypi
blas                      1.0                         mkl  
blessed                   1.20.0                   pypi_0    pypi
blobfile                  3.0.0                    pypi_0    pypi
build                     1.2.2.post1              pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6  
ca-certificates           2022.9.24            ha878542_0    conda-forge
certifi                   2022.12.7                pypi_0    pypi
charset-normalizer        2.1.1                    pypi_0    pypi
cpufeature                0.2.1                    pypi_0    pypi
fastapi                   0.115.8                  pypi_0    pypi
filelock                  3.13.1                   pypi_0    pypi
flash-attn                2.7.4.post1              pypi_0    pypi
frozenlist                1.5.0                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
intel-openmp              2023.1.0         hdb19cb5_46306  
jinja2                    3.1.4                    pypi_0    pypi
jsonpatch                 1.33                     pypi_0    pypi
jsonpointer               3.0.0                    pypi_0    pypi
ktransformers             0.2.2rc1+cu124torch24avx2          pypi_0    pypi
langchain                 0.3.19                   pypi_0    pypi
langchain-core            0.3.40                   pypi_0    pypi
langchain-text-splitters  0.3.6                    pypi_0    pypi
langsmith                 0.3.11                   pypi_0    pypi
ld_impl_linux-64          2.40                 h12ee557_0  
libabseil                 20240116.2      cxx17_h6a678d5_0  
libffi                    3.4.4                h6a678d5_1  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libprotobuf               4.25.3               he621ea3_0  
libsentencepiece          0.2.0                h70df78d_2  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
markupsafe                2.1.5                    pypi_0    pypi
mkl                       2023.1.0         h213fc3f_46344  
mkl-fft                   1.3.11                   pypi_0    pypi
mkl-random                1.2.8                    pypi_0    pypi
mkl-service               2.4.0                    pypi_0    pypi
mkl_fft                   1.3.11          py311h5eee18b_0  
mkl_random                1.2.8           py311ha02d727_0  
multidict                 6.1.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
numpy                     1.26.4                   pypi_0    pypi
numpy-base                2.2.2           py311hf175353_0  
nvidia-cublas-cu12        12.4.2.65                pypi_0    pypi
nvidia-cuda-cupti-cu12    12.4.99                  pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.4.99                  pypi_0    pypi
nvidia-cuda-runtime-cu12  12.4.99                  pypi_0    pypi
nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
nvidia-cufft-cu12         11.2.0.44                pypi_0    pypi
nvidia-curand-cu12        10.3.5.119               pypi_0    pypi
nvidia-cusolver-cu12      11.6.0.99                pypi_0    pypi
nvidia-cusparse-cu12      12.3.0.142               pypi_0    pypi
nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.4.99                  pypi_0    pypi
nvidia-nvtx-cu12          12.4.99                  pypi_0    pypi
openssl                   3.0.15               h5eee18b_0  
orjson                    3.10.15                  pypi_0    pypi
packaging                 24.2                     pypi_0    pypi
pip                       25.0                     pypi_0    pypi
propcache                 0.3.0                    pypi_0    pypi
psutil                    7.0.0                    pypi_0    pypi
pycryptodomex             3.21.0                   pypi_0    pypi
pydantic                  2.10.6                   pypi_0    pypi
pydantic-core             2.27.2                   pypi_0    pypi
pyproject-hooks           1.2.0                    pypi_0    pypi
python                    3.11.11              he870216_0  
pyyaml                    6.0.2                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
regex                     2024.11.6                pypi_0    pypi
requests                  2.28.1                   pypi_0    pypi
requests-toolbelt         1.0.0                    pypi_0    pypi
safetensors               0.5.2                    pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
sentencepiece-python      0.2.0           py311h3e0de92_2  
sentencepiece-spm         0.2.0                h70df78d_2  
setuptools                72.1.0                   pypi_0    pypi
six                       1.17.0                   pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
sqlite                    3.45.3               h5eee18b_0  
starlette                 0.45.3                   pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
tbb                       2021.8.0             hdb19cb5_0  
termcolor                 2.5.0                    pypi_0    pypi
tk                        8.6.14               h39e8969_0  
tokenizers                0.19.1                   pypi_0    pypi
torch                     2.4.0+cu124              pypi_0    pypi
torchaudio                2.4.0+cu124              pypi_0    pypi
torchvision               0.19.0+cu124             pypi_0    pypi
tqdm                      4.67.1                   pypi_0    pypi
transformers              4.43.2                   pypi_0    pypi
triton                    3.0.0                    pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2025a                h04d1e81_0  
uvicorn                   0.34.0                   pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
wheel                     0.45.1                   pypi_0    pypi
xz                        5.6.4                h5eee18b_1  
yarl                      1.18.3                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_1  
zstandard                 0.23.0                   pypi_0    pypi

Originally created by @xiaohangguo on GitHub (Feb 26, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/693 您好，我按照官方给出的教程https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/install.md 配置了环境，并且正常跑起来了。但是显卡的利用率始终为0，感觉是有问题的。推理全靠cpu，token吐的速度很慢很慢。 加载完模型花费了一个小时。然后大概四五分钟推理出一个token。 运行时会输出一堆的： ``` “for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows” ``` 详细见后面附件中的图 一些背景信息： 服务器：centos8 内存：256G cpus core ：40 Model name: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz GPU：4090 48G  运行时信息：  环境版本如下： ``` lvshuhang@s04:~ $ conda list # packages in environment at /home/lvshuhang/.conda/envs/ktransformers: # # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu accelerate 1.4.0 pypi_0 pypi aiohappyeyeballs 2.4.6 pypi_0 pypi aiohttp 3.11.13 pypi_0 pypi aiosignal 1.3.2 pypi_0 pypi annotated-types 0.7.0 pypi_0 pypi anyio 4.8.0 pypi_0 pypi attrs 25.1.0 pypi_0 pypi blas 1.0 mkl blessed 1.20.0 pypi_0 pypi blobfile 3.0.0 pypi_0 pypi build 1.2.2.post1 pypi_0 pypi bzip2 1.0.8 h5eee18b_6 ca-certificates 2022.9.24 ha878542_0 conda-forge certifi 2022.12.7 pypi_0 pypi charset-normalizer 2.1.1 pypi_0 pypi cpufeature 0.2.1 pypi_0 pypi fastapi 0.115.8 pypi_0 pypi filelock 3.13.1 pypi_0 pypi flash-attn 2.7.4.post1 pypi_0 pypi frozenlist 1.5.0 pypi_0 pypi idna 3.4 pypi_0 pypi intel-openmp 2023.1.0 hdb19cb5_46306 jinja2 3.1.4 pypi_0 pypi jsonpatch 1.33 pypi_0 pypi jsonpointer 3.0.0 pypi_0 pypi ktransformers 0.2.2rc1+cu124torch24avx2 pypi_0 pypi langchain 0.3.19 pypi_0 pypi langchain-core 0.3.40 pypi_0 pypi langchain-text-splitters 0.3.6 pypi_0 pypi langsmith 0.3.11 pypi_0 pypi ld_impl_linux-64 2.40 h12ee557_0 libabseil 20240116.2 cxx17_h6a678d5_0 libffi 3.4.4 h6a678d5_1 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libprotobuf 4.25.3 he621ea3_0 libsentencepiece 0.2.0 h70df78d_2 libstdcxx-ng 11.2.0 h1234567_1 libuuid 1.41.5 h5eee18b_0 markupsafe 2.1.5 pypi_0 pypi mkl 2023.1.0 h213fc3f_46344 mkl-fft 1.3.11 pypi_0 pypi mkl-random 1.2.8 pypi_0 pypi mkl-service 2.4.0 pypi_0 pypi mkl_fft 1.3.11 py311h5eee18b_0 mkl_random 1.2.8 py311ha02d727_0 multidict 6.1.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 numpy 1.26.4 pypi_0 pypi numpy-base 2.2.2 py311hf175353_0 nvidia-cublas-cu12 12.4.2.65 pypi_0 pypi nvidia-cuda-cupti-cu12 12.4.99 pypi_0 pypi nvidia-cuda-nvrtc-cu12 12.4.99 pypi_0 pypi nvidia-cuda-runtime-cu12 12.4.99 pypi_0 pypi nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi nvidia-cufft-cu12 11.2.0.44 pypi_0 pypi nvidia-curand-cu12 10.3.5.119 pypi_0 pypi nvidia-cusolver-cu12 11.6.0.99 pypi_0 pypi nvidia-cusparse-cu12 12.3.0.142 pypi_0 pypi nvidia-nccl-cu12 2.20.5 pypi_0 pypi nvidia-nvjitlink-cu12 12.4.99 pypi_0 pypi nvidia-nvtx-cu12 12.4.99 pypi_0 pypi openssl 3.0.15 h5eee18b_0 orjson 3.10.15 pypi_0 pypi packaging 24.2 pypi_0 pypi pip 25.0 pypi_0 pypi propcache 0.3.0 pypi_0 pypi psutil 7.0.0 pypi_0 pypi pycryptodomex 3.21.0 pypi_0 pypi pydantic 2.10.6 pypi_0 pypi pydantic-core 2.27.2 pypi_0 pypi pyproject-hooks 1.2.0 pypi_0 pypi python 3.11.11 he870216_0 pyyaml 6.0.2 pypi_0 pypi readline 8.2 h5eee18b_0 regex 2024.11.6 pypi_0 pypi requests 2.28.1 pypi_0 pypi requests-toolbelt 1.0.0 pypi_0 pypi safetensors 0.5.2 pypi_0 pypi sentencepiece 0.2.0 pypi_0 pypi sentencepiece-python 0.2.0 py311h3e0de92_2 sentencepiece-spm 0.2.0 h70df78d_2 setuptools 72.1.0 pypi_0 pypi six 1.17.0 pypi_0 pypi sniffio 1.3.1 pypi_0 pypi sqlite 3.45.3 h5eee18b_0 starlette 0.45.3 pypi_0 pypi sympy 1.13.1 pypi_0 pypi tbb 2021.8.0 hdb19cb5_0 termcolor 2.5.0 pypi_0 pypi tk 8.6.14 h39e8969_0 tokenizers 0.19.1 pypi_0 pypi torch 2.4.0+cu124 pypi_0 pypi torchaudio 2.4.0+cu124 pypi_0 pypi torchvision 0.19.0+cu124 pypi_0 pypi tqdm 4.67.1 pypi_0 pypi transformers 4.43.2 pypi_0 pypi triton 3.0.0 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tzdata 2025a h04d1e81_0 uvicorn 0.34.0 pypi_0 pypi wcwidth 0.2.13 pypi_0 pypi wheel 0.45.1 pypi_0 pypi xz 5.6.4 h5eee18b_1 yarl 1.18.3 pypi_0 pypi zlib 1.2.13 h5eee18b_1 zstandard 0.23.0 pypi_0 pypi ```

bsakaguchi closed this issue 2026-10-29 14:40:36 +00:00

bsakaguchi commented 2026-10-29 14:40:36 +00:00

Author

Owner

Copy link

@ubergarm commented on GitHub (Feb 27, 2025):

@xiaohangguo

查看您在另一工单 #641 提供的命令，似乎未包含 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml。您需要编辑该 .yaml 文件，或选择与您的GPU配置及CUDA设备编号匹配的配置文件。

参考此指南查看完整命令示例及讨论。祝顺利！

---

xiaohangguo

Looking at your command that you gave in a different ticket #641 it seems like you did not include --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml. You must edit that .yaml file or choose one that matches your GPU setup and CUDA device numbers.

Check out this guide for full command examples and discussion. Good luck!

<!-- gh-comment-id:2688381584 --> @ubergarm commented on GitHub (Feb 27, 2025): @xiaohangguo 查看您在另一工单 #641 提供的命令，似乎未包含 `--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml`。您需要编辑该 `.yaml` 文件，或选择与您的GPU配置及CUDA设备编号匹配的配置文件。 参考[此指南查看完整命令示例及讨论](https://github.com/ubergarm/r1-ktransformers-guide/blob/main/README.zh.md)。祝顺利！ --- xiaohangguo Looking at your command that you gave in a different ticket #641 it seems like you did not include `--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml`. You must edit that `.yaml` file or choose one that matches your GPU setup and CUDA device numbers. Check out [this guide for full command examples and discussion](https://github.com/ubergarm/r1-ktransformers-guide/). Good luck!

bsakaguchi commented 2026-10-29 14:40:37 +00:00

Author

Owner

Copy link

@xiaohangguo commented on GitHub (Feb 28, 2025):

没有用。加了优化器相关设置之后，GPU的利用率仍然是0，相当于推理都用的是cpu。不知道怎么解决
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/lvshuhang/.cache/modelscope/hub/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --hf_token hf_*** --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --cpu_infer 24

Chat: who are you
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
for Windows or GPU before ampere, use forward_windows
Greetings! I'm 

<!-- gh-comment-id:2689600072 --> @xiaohangguo commented on GitHub (Feb 28, 2025): 没有用。加了优化器相关设置之后，GPU的利用率仍然是0，相当于推理都用的是cpu。不知道怎么解决 python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/lvshuhang/.cache/modelscope/hub/models/unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --hf_token hf_*** --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --cpu_infer 24 ``` Chat: who are you for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows for Windows or GPU before ampere, use forward_windows Greetings! I'm ```

bsakaguchi commented 2026-10-29 14:40:37 +00:00

Author

Owner

Copy link

@Azure-Tang commented on GitHub (Feb 28, 2025):

改过yaml么，load的时候输出的日志显示是load到哪个device上了呢？

您具体的decode和prefill速度是多少；可以和ollama进行一次对比，如果确实是ollama的2～3倍说明只是您的驱动显示不准，换一个统计工具试一下。

<!-- gh-comment-id:2690540975 --> @Azure-Tang commented on GitHub (Feb 28, 2025): 改过yaml么，load的时候输出的日志显示是load到哪个device上了呢？ 您具体的decode和prefill速度是多少；可以和ollama进行一次对比，如果确实是ollama的2～3倍说明只是您的驱动显示不准，换一个统计工具试一下。

bsakaguchi commented 2026-10-29 14:40:37 +00:00

Author

Owner

Copy link

@xiaohangguo commented on GitHub (Feb 28, 2025):

改过yaml么，load的时候输出的日志显示是load到哪个device上了呢？

您具体的decode和prefill速度是多少；可以和ollama进行一次对比，如果确实是ollama的2～3倍说明只是您的驱动显示不准，换一个统计工具试一下。

没有改yaml，就用的官方给出的V3的默认配置。日志是load到cuda:0,也就是这里的4090上。
您具体的decode和prefill速度是多少，这个我没测，但我感知上面就是纯cpu在跑，我一直在看GPU的利用率，始终是0，然后大概1-2分钟才出来一个token。
另外我从0~60层全部加载完要花费一个小时，不知道这个时间是否合理。
我怀疑就是GPU没有工作，但是不知道怎么debug。我可以尝试用小模型来debug，您有好的debug方案吗？

<!-- gh-comment-id:2690658195 --> @xiaohangguo commented on GitHub (Feb 28, 2025): > 改过yaml么，load的时候输出的日志显示是load到哪个device上了呢？ > > 您具体的decode和prefill速度是多少；可以和ollama进行一次对比，如果确实是ollama的2～3倍说明只是您的驱动显示不准，换一个统计工具试一下。 没有改yaml，就用的官方给出的V3的默认配置。日志是load到cuda:0,也就是这里的4090上。 您具体的decode和prefill速度是多少，这个我没测，但我感知上面就是纯cpu在跑，我一直在看GPU的利用率，始终是0，然后大概1-2分钟才出来一个token。 另外我从0~60层全部加载完要花费一个小时，不知道这个时间是否合理。 我怀疑就是GPU没有工作，但是不知道怎么debug。我可以尝试用小模型来debug，您有好的debug方案吗？

bsakaguchi commented 2026-10-29 14:40:37 +00:00

Author

Owner

Copy link

@ubergarm commented on GitHub (Feb 28, 2025):

@xiaohangguo

内存：256G

你指望在256GB系统内存中运行unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M（377GB模型）能怎样？它居然能运行已经是mmap()的 功劳了。瓶颈在于你的磁盘驱动页缓存缓冲I/O——毕竟GPU根本无事可做，只能干等最高不过几GB/s的磁盘读写。

可用btop或iotop等工具检查磁盘I/O。

建议改用UD-Q2_K_XL量化版（212GB模型），它能轻松载入内存，在保证质量的同时获得可用速度。我自己就在用。

祝好运！

---

xiaohangguo

System RAM: 256GB

What do you expect trying to run unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M 377GB model size in 256GB system RAM? It is amazing it works at all thanks to mmap(). The bottleneck is your disk drive page cache buffered i/o. Of course the GPU is not doing any work waiting on a few GB/s disk drive at best.

You can check disk i/o with btop or iotop etc.

I advise you use the UD-Q2_K_XL quant at 212 GB model size it will fit comfortably in memory and you will get a usable speed with good quality. I use it myself.

Good luck!

<!-- gh-comment-id:2691265533 --> @ubergarm commented on GitHub (Feb 28, 2025): @xiaohangguo > 内存：256G 你指望在256GB系统内存中运行`unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M`（377GB模型）能怎样？它居然能运行已经是`mmap()`的 功劳了。瓶颈在于你的磁盘驱动页缓存缓冲I/O——毕竟GPU根本无事可做，只能干等最高不过几GB/s的磁盘读写。 可用`btop`或`iotop`等工具检查磁盘I/O。 建议改用`UD-Q2_K_XL`量化版（212GB模型），它能轻松载入内存，在保证质量的同时获得可用速度。我自己就在用。 祝好运！ --- xiaohangguo > System RAM: 256GB What do you expect trying to run `unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M` 377GB model size in 256GB system RAM? It is amazing it works at all thanks to `mmap()`. The bottleneck is your disk drive page cache buffered i/o. Of course the GPU is not doing any work waiting on a few GB/s disk drive at best. You can check disk i/o with `btop` or `iotop` etc. I advise you use the `UD-Q2_K_XL` quant at 212 GB model size it will fit comfortably in memory and you will get a usable speed with good quality. I use it myself. Good luck!

bsakaguchi commented 2026-10-29 14:40:37 +00:00

Author

Owner

Copy link

@xiaohangguo commented on GitHub (Mar 12, 2025):

我把内存加大了，现在成功了。

<!-- gh-comment-id:2717203828 --> @xiaohangguo commented on GitHub (Mar 12, 2025): 我把内存加大了，现在成功了。 

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#505

No description provided.