kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #909] [Bug] Rocm run DeepSeek-R1-UD-IQ1_S failed #649

New issue
Closed
opened 2026-10-29 14:42:25 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-29 14:42:25 +00:00
Owner
Copy link

Originally created by @Ryuukeisyou on GitHub (Mar 16, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/909

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
  • 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

Hello and thank you for your work!

I built ktransformers and tried server times to run DeepSeek-R1-UD-IQ1_S, there is only one time it generated several tokens (very slow, much slower than using llma.cpp) and then crash, and most of the time it crashed without generating anything.

Output:

loading output to cuda using CPUInfer
loading token_embd.weight to cpu
loading token_embd.weight with CPU
loading blk.0.attn_q_a.weight with CPU
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_q_b.weight with CPU
loading blk.0.attn_kv_a_mqa.weight with CPU
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
loading blk.0.attn_output.weight with CPU
loading blk.0.ffn_gate.weight with CPU
loading blk.0.ffn_up.weight with CPU
loading blk.0.ffn_down.weight with CPU
loading blk.0.attn_norm.weight to cuda:0
loading blk.0.ffn_norm.weight to cuda:0
loading blk.1.attn_q_a.weight with CPU
loading blk.1.attn_q_a_norm.weight to cuda:0
loading blk.1.attn_q_b.weight with CPU
loading blk.1.attn_kv_a_mqa.weight with CPU
loading blk.1.attn_kv_a_norm.weight to cuda:0
loading blk.1.attn_kv_b.weight to cuda:0
loading blk.1.attn_output.weight with CPU
loading blk.1.ffn_gate.weight with CPU
loading blk.1.ffn_up.weight with CPU
loading blk.1.ffn_down.weight with CPU
loading blk.1.attn_norm.weight to cuda:0
loading blk.1.ffn_norm.weight to cuda:0
loading blk.2.attn_q_a.weight with CPU
loading blk.2.attn_q_a_norm.weight to cuda:0
loading blk.2.attn_q_b.weight with CPU
loading blk.2.attn_kv_a_mqa.weight with CPU
loading blk.2.attn_kv_a_norm.weight to cuda:0
loading blk.2.attn_kv_b.weight to cuda:0
loading blk.2.attn_output.weight with CPU
loading blk.2.ffn_gate.weight with CPU
loading blk.2.ffn_up.weight with CPU
loading blk.2.ffn_down.weight with CPU
loading blk.2.attn_norm.weight to cuda:0
loading blk.2.ffn_norm.weight to cuda:0
loading blk.3.attn_q_a.weight with CPU
loading blk.3.attn_q_a_norm.weight to cuda:0
loading blk.3.attn_q_b.weight with CPU
loading blk.3.attn_kv_a_mqa.weight with CPU
loading blk.3.attn_kv_a_norm.weight to cuda:0
loading blk.3.attn_kv_b.weight to cuda:0
loading blk.3.attn_output.weight with CPU
loading blk.3.ffn_gate_shexp.weight with CPU
loading blk.3.ffn_up_shexp.weight with CPU
loading blk.3.ffn_down_shexp.weight with CPU
loading blk.3.attn_norm.weight to cuda:0
loading blk.3.ffn_norm.weight to cuda:0
loading blk.4.attn_q_a.weight with CPU
loading blk.4.attn_q_a_norm.weight to cuda:0
loading blk.4.attn_q_b.weight with CPU
loading blk.4.attn_kv_a_mqa.weight with CPU
loading blk.4.attn_kv_a_norm.weight to cuda:0
loading blk.4.attn_kv_b.weight to cuda:0
loading blk.4.attn_output.weight with CPU
loading blk.4.ffn_gate_shexp.weight with CPU
loading blk.4.ffn_up_shexp.weight with CPU
loading blk.4.ffn_down_shexp.weight with CPU
loading blk.4.attn_norm.weight to cuda:0
loading blk.4.ffn_norm.weight to cuda:0
loading blk.5.attn_q_a.weight with CPU
loading blk.5.attn_q_a_norm.weight to cuda:0
loading blk.5.attn_q_b.weight with CPU
loading blk.5.attn_kv_a_mqa.weight with CPU
loading blk.5.attn_kv_a_norm.weight to cuda:0
loading blk.5.attn_kv_b.weight to cuda:0
loading blk.5.attn_output.weight with CPU
loading blk.5.ffn_gate_shexp.weight with CPU
loading blk.5.ffn_up_shexp.weight with CPU
loading blk.5.ffn_down_shexp.weight with CPU
loading blk.5.attn_norm.weight to cuda:0
loading blk.5.ffn_norm.weight to cuda:0
loading blk.6.attn_q_a.weight with CPU
loading blk.6.attn_q_a_norm.weight to cuda:0
loading blk.6.attn_q_b.weight with CPU
loading blk.6.attn_kv_a_mqa.weight with CPU
loading blk.6.attn_kv_a_norm.weight to cuda:0
loading blk.6.attn_kv_b.weight to cuda:0
loading blk.6.attn_output.weight with CPU
loading blk.6.ffn_gate_shexp.weight with CPU
loading blk.6.ffn_up_shexp.weight with CPU
loading blk.6.ffn_down_shexp.weight with CPU
loading blk.6.attn_norm.weight to cuda:0
loading blk.6.ffn_norm.weight to cuda:0
loading blk.7.attn_q_a.weight with CPU
loading blk.7.attn_q_a_norm.weight to cuda:0
loading blk.7.attn_q_b.weight with CPU
loading blk.7.attn_kv_a_mqa.weight with CPU
loading blk.7.attn_kv_a_norm.weight to cuda:0
loading blk.7.attn_kv_b.weight to cuda:0
loading blk.7.attn_output.weight with CPU
loading blk.7.ffn_gate_shexp.weight with CPU
loading blk.7.ffn_up_shexp.weight with CPU
loading blk.7.ffn_down_shexp.weight with CPU
loading blk.7.attn_norm.weight to cuda:0
loading blk.7.ffn_norm.weight to cuda:0
loading blk.8.attn_q_a.weight with CPU
loading blk.8.attn_q_a_norm.weight to cuda:0
loading blk.8.attn_q_b.weight with CPU
loading blk.8.attn_kv_a_mqa.weight with CPU
loading blk.8.attn_kv_a_norm.weight to cuda:0
loading blk.8.attn_kv_b.weight to cuda:0
loading blk.8.attn_output.weight with CPU
loading blk.8.ffn_gate_shexp.weight with CPU
loading blk.8.ffn_up_shexp.weight with CPU
loading blk.8.ffn_down_shexp.weight with CPU
loading blk.8.attn_norm.weight to cuda:0
loading blk.8.ffn_norm.weight to cuda:0
loading blk.9.attn_q_a.weight with CPU
loading blk.9.attn_q_a_norm.weight to cuda:0
loading blk.9.attn_q_b.weight with CPU
loading blk.9.attn_kv_a_mqa.weight with CPU
loading blk.9.attn_kv_a_norm.weight to cuda:0
loading blk.9.attn_kv_b.weight to cuda:0
loading blk.9.attn_output.weight with CPU
loading blk.9.ffn_gate_shexp.weight with CPU
loading blk.9.ffn_up_shexp.weight with CPU
loading blk.9.ffn_down_shexp.weight with CPU
loading blk.9.attn_norm.weight to cuda:0
loading blk.9.ffn_norm.weight to cuda:0
loading blk.10.attn_q_a.weight with CPU
loading blk.10.attn_q_a_norm.weight to cuda:0
loading blk.10.attn_q_b.weight with CPU
loading blk.10.attn_kv_a_mqa.weight with CPU
loading blk.10.attn_kv_a_norm.weight to cuda:0
loading blk.10.attn_kv_b.weight to cuda:0
loading blk.10.attn_output.weight with CPU
loading blk.10.ffn_gate_shexp.weight with CPU
loading blk.10.ffn_up_shexp.weight with CPU
loading blk.10.ffn_down_shexp.weight with CPU
loading blk.10.attn_norm.weight to cuda:0
loading blk.10.ffn_norm.weight to cuda:0
loading blk.11.attn_q_a.weight with CPU
loading blk.11.attn_q_a_norm.weight to cuda:0
loading blk.11.attn_q_b.weight with CPU
loading blk.11.attn_kv_a_mqa.weight with CPU
loading blk.11.attn_kv_a_norm.weight to cuda:0
loading blk.11.attn_kv_b.weight to cuda:0
loading blk.11.attn_output.weight with CPU
loading blk.11.ffn_gate_shexp.weight with CPU
loading blk.11.ffn_up_shexp.weight with CPU
loading blk.11.ffn_down_shexp.weight with CPU
loading blk.11.attn_norm.weight to cuda:0
loading blk.11.ffn_norm.weight to cuda:0
loading blk.12.attn_q_a.weight with CPU
loading blk.12.attn_q_a_norm.weight to cuda:0
loading blk.12.attn_q_b.weight with CPU
loading blk.12.attn_kv_a_mqa.weight with CPU
loading blk.12.attn_kv_a_norm.weight to cuda:0
loading blk.12.attn_kv_b.weight to cuda:0
loading blk.12.attn_output.weight with CPU
loading blk.12.ffn_gate_shexp.weight with CPU
loading blk.12.ffn_up_shexp.weight with CPU
loading blk.12.ffn_down_shexp.weight with CPU
loading blk.12.attn_norm.weight to cuda:0
loading blk.12.ffn_norm.weight to cuda:0
loading blk.13.attn_q_a.weight with CPU
loading blk.13.attn_q_a_norm.weight to cuda:0
loading blk.13.attn_q_b.weight with CPU
loading blk.13.attn_kv_a_mqa.weight with CPU
loading blk.13.attn_kv_a_norm.weight to cuda:0
loading blk.13.attn_kv_b.weight to cuda:0
loading blk.13.attn_output.weight with CPU
loading blk.13.ffn_gate_shexp.weight with CPU
loading blk.13.ffn_up_shexp.weight with CPU
loading blk.13.ffn_down_shexp.weight with CPU
loading blk.13.attn_norm.weight to cuda:0
loading blk.13.ffn_norm.weight to cuda:0
loading blk.14.attn_q_a.weight with CPU
loading blk.14.attn_q_a_norm.weight to cuda:0
loading blk.14.attn_q_b.weight with CPU
loading blk.14.attn_kv_a_mqa.weight with CPU
loading blk.14.attn_kv_a_norm.weight to cuda:0
loading blk.14.attn_kv_b.weight to cuda:0
loading blk.14.attn_output.weight with CPU
loading blk.14.ffn_gate_shexp.weight with CPU
loading blk.14.ffn_up_shexp.weight with CPU
loading blk.14.ffn_down_shexp.weight with CPU
loading blk.14.attn_norm.weight to cuda:0
loading blk.14.ffn_norm.weight to cuda:0
loading blk.15.attn_q_a.weight with CPU
loading blk.15.attn_q_a_norm.weight to cuda:0
loading blk.15.attn_q_b.weight with CPU
loading blk.15.attn_kv_a_mqa.weight with CPU
loading blk.15.attn_kv_a_norm.weight to cuda:0
loading blk.15.attn_kv_b.weight to cuda:0
loading blk.15.attn_output.weight with CPU
loading blk.15.ffn_gate_shexp.weight with CPU
loading blk.15.ffn_up_shexp.weight with CPU
loading blk.15.ffn_down_shexp.weight with CPU
loading blk.15.attn_norm.weight to cuda:0
loading blk.15.ffn_norm.weight to cuda:0
loading blk.16.attn_q_a.weight with CPU
loading blk.16.attn_q_a_norm.weight to cuda:0
loading blk.16.attn_q_b.weight with CPU
loading blk.16.attn_kv_a_mqa.weight with CPU
loading blk.16.attn_kv_a_norm.weight to cuda:0
loading blk.16.attn_kv_b.weight to cuda:0
loading blk.16.attn_output.weight with CPU
loading blk.16.ffn_gate_shexp.weight with CPU
loading blk.16.ffn_up_shexp.weight with CPU
loading blk.16.ffn_down_shexp.weight with CPU
loading blk.16.attn_norm.weight to cuda:0
loading blk.16.ffn_norm.weight to cuda:0
loading blk.17.attn_q_a.weight with CPU
loading blk.17.attn_q_a_norm.weight to cuda:0
loading blk.17.attn_q_b.weight with CPU
loading blk.17.attn_kv_a_mqa.weight with CPU
loading blk.17.attn_kv_a_norm.weight to cuda:0
loading blk.17.attn_kv_b.weight to cuda:0
loading blk.17.attn_output.weight with CPU
loading blk.17.ffn_gate_shexp.weight with CPU
loading blk.17.ffn_up_shexp.weight with CPU
loading blk.17.ffn_down_shexp.weight with CPU
loading blk.17.attn_norm.weight to cuda:0
loading blk.17.ffn_norm.weight to cuda:0
loading blk.18.attn_q_a.weight with CPU
loading blk.18.attn_q_a_norm.weight to cuda:0
loading blk.18.attn_q_b.weight with CPU
loading blk.18.attn_kv_a_mqa.weight with CPU
loading blk.18.attn_kv_a_norm.weight to cuda:0
loading blk.18.attn_kv_b.weight to cuda:0
loading blk.18.attn_output.weight with CPU
loading blk.18.ffn_gate_shexp.weight with CPU
loading blk.18.ffn_up_shexp.weight with CPU
loading blk.18.ffn_down_shexp.weight with CPU
loading blk.18.attn_norm.weight to cuda:0
loading blk.18.ffn_norm.weight to cuda:0
loading blk.19.attn_q_a.weight with CPU
loading blk.19.attn_q_a_norm.weight to cuda:0
loading blk.19.attn_q_b.weight with CPU
loading blk.19.attn_kv_a_mqa.weight with CPU
loading blk.19.attn_kv_a_norm.weight to cuda:0
loading blk.19.attn_kv_b.weight to cuda:0
loading blk.19.attn_output.weight with CPU
loading blk.19.ffn_gate_shexp.weight with CPU
loading blk.19.ffn_up_shexp.weight with CPU
loading blk.19.ffn_down_shexp.weight with CPU
loading blk.19.attn_norm.weight to cuda:0
loading blk.19.ffn_norm.weight to cuda:0
loading blk.20.attn_q_a.weight with CPU
loading blk.20.attn_q_a_norm.weight to cuda:0
loading blk.20.attn_q_b.weight with CPU
loading blk.20.attn_kv_a_mqa.weight with CPU
loading blk.20.attn_kv_a_norm.weight to cuda:0
loading blk.20.attn_kv_b.weight to cuda:0
loading blk.20.attn_output.weight with CPU
loading blk.20.ffn_gate_shexp.weight with CPU
loading blk.20.ffn_up_shexp.weight with CPU
loading blk.20.ffn_down_shexp.weight with CPU
loading blk.20.attn_norm.weight to cuda:0
loading blk.20.ffn_norm.weight to cuda:0
loading blk.21.attn_q_a.weight with CPU
loading blk.21.attn_q_a_norm.weight to cuda:0
loading blk.21.attn_q_b.weight with CPU
loading blk.21.attn_kv_a_mqa.weight with CPU
loading blk.21.attn_kv_a_norm.weight to cuda:0
loading blk.21.attn_kv_b.weight to cuda:0
loading blk.21.attn_output.weight with CPU
loading blk.21.ffn_gate_shexp.weight with CPU
loading blk.21.ffn_up_shexp.weight with CPU
loading blk.21.ffn_down_shexp.weight with CPU
loading blk.21.attn_norm.weight to cuda:0
loading blk.21.ffn_norm.weight to cuda:0
loading blk.22.attn_q_a.weight with CPU
loading blk.22.attn_q_a_norm.weight to cuda:0
loading blk.22.attn_q_b.weight with CPU
loading blk.22.attn_kv_a_mqa.weight with CPU
loading blk.22.attn_kv_a_norm.weight to cuda:0
loading blk.22.attn_kv_b.weight to cuda:0
loading blk.22.attn_output.weight with CPU
loading blk.22.ffn_gate_shexp.weight with CPU
loading blk.22.ffn_up_shexp.weight with CPU
loading blk.22.ffn_down_shexp.weight with CPU
loading blk.22.attn_norm.weight to cuda:0
loading blk.22.ffn_norm.weight to cuda:0
loading blk.23.attn_q_a.weight with CPU
loading blk.23.attn_q_a_norm.weight to cuda:0
loading blk.23.attn_q_b.weight with CPU
loading blk.23.attn_kv_a_mqa.weight with CPU
loading blk.23.attn_kv_a_norm.weight to cuda:0
loading blk.23.attn_kv_b.weight to cuda:0
loading blk.23.attn_output.weight with CPU
loading blk.23.ffn_gate_shexp.weight with CPU
loading blk.23.ffn_up_shexp.weight with CPU
loading blk.23.ffn_down_shexp.weight with CPU
loading blk.23.attn_norm.weight to cuda:0
loading blk.23.ffn_norm.weight to cuda:0
loading blk.24.attn_q_a.weight with CPU
loading blk.24.attn_q_a_norm.weight to cuda:0
loading blk.24.attn_q_b.weight with CPU
loading blk.24.attn_kv_a_mqa.weight with CPU
loading blk.24.attn_kv_a_norm.weight to cuda:0
loading blk.24.attn_kv_b.weight to cuda:0
loading blk.24.attn_output.weight with CPU
loading blk.24.ffn_gate_shexp.weight with CPU
loading blk.24.ffn_up_shexp.weight with CPU
loading blk.24.ffn_down_shexp.weight with CPU
loading blk.24.attn_norm.weight to cuda:0
loading blk.24.ffn_norm.weight to cuda:0
loading blk.25.attn_q_a.weight with CPU
loading blk.25.attn_q_a_norm.weight to cuda:0
loading blk.25.attn_q_b.weight with CPU
loading blk.25.attn_kv_a_mqa.weight with CPU
loading blk.25.attn_kv_a_norm.weight to cuda:0
loading blk.25.attn_kv_b.weight to cuda:0
loading blk.25.attn_output.weight with CPU
loading blk.25.ffn_gate_shexp.weight with CPU
loading blk.25.ffn_up_shexp.weight with CPU
loading blk.25.ffn_down_shexp.weight with CPU
loading blk.25.attn_norm.weight to cuda:0
loading blk.25.ffn_norm.weight to cuda:0
loading blk.26.attn_q_a.weight with CPU
loading blk.26.attn_q_a_norm.weight to cuda:0
loading blk.26.attn_q_b.weight with CPU
loading blk.26.attn_kv_a_mqa.weight with CPU
loading blk.26.attn_kv_a_norm.weight to cuda:0
loading blk.26.attn_kv_b.weight to cuda:0
loading blk.26.attn_output.weight with CPU
loading blk.26.ffn_gate_shexp.weight with CPU
loading blk.26.ffn_up_shexp.weight with CPU
loading blk.26.ffn_down_shexp.weight with CPU
loading blk.26.attn_norm.weight to cuda:0
loading blk.26.ffn_norm.weight to cuda:0
loading blk.27.attn_q_a.weight with CPU
loading blk.27.attn_q_a_norm.weight to cuda:0
loading blk.27.attn_q_b.weight with CPU
loading blk.27.attn_kv_a_mqa.weight with CPU
loading blk.27.attn_kv_a_norm.weight to cuda:0
loading blk.27.attn_kv_b.weight to cuda:0
loading blk.27.attn_output.weight with CPU
loading blk.27.ffn_gate_shexp.weight with CPU
loading blk.27.ffn_up_shexp.weight with CPU
loading blk.27.ffn_down_shexp.weight with CPU
loading blk.27.attn_norm.weight to cuda:0
loading blk.27.ffn_norm.weight to cuda:0
loading blk.28.attn_q_a.weight with CPU
loading blk.28.attn_q_a_norm.weight to cuda:0
loading blk.28.attn_q_b.weight with CPU
loading blk.28.attn_kv_a_mqa.weight with CPU
loading blk.28.attn_kv_a_norm.weight to cuda:0
loading blk.28.attn_kv_b.weight to cuda:0
loading blk.28.attn_output.weight with CPU
loading blk.28.ffn_gate_shexp.weight with CPU
loading blk.28.ffn_up_shexp.weight with CPU
loading blk.28.ffn_down_shexp.weight with CPU
loading blk.28.attn_norm.weight to cuda:0
loading blk.28.ffn_norm.weight to cuda:0
loading blk.29.attn_q_a.weight with CPU
loading blk.29.attn_q_a_norm.weight to cuda:0
loading blk.29.attn_q_b.weight with CPU
loading blk.29.attn_kv_a_mqa.weight with CPU
loading blk.29.attn_kv_a_norm.weight to cuda:0
loading blk.29.attn_kv_b.weight to cuda:0
loading blk.29.attn_output.weight with CPU
loading blk.29.ffn_gate_shexp.weight with CPU
loading blk.29.ffn_up_shexp.weight with CPU
loading blk.29.ffn_down_shexp.weight with CPU
loading blk.29.attn_norm.weight to cuda:0
loading blk.29.ffn_norm.weight to cuda:0
loading blk.30.attn_q_a.weight with CPU
loading blk.30.attn_q_a_norm.weight to cuda:0
loading blk.30.attn_q_b.weight with CPU
loading blk.30.attn_kv_a_mqa.weight with CPU
loading blk.30.attn_kv_a_norm.weight to cuda:0
loading blk.30.attn_kv_b.weight to cuda:0
loading blk.30.attn_output.weight with CPU
loading blk.30.ffn_gate_shexp.weight with CPU
loading blk.30.ffn_up_shexp.weight with CPU
loading blk.30.ffn_down_shexp.weight with CPU
loading blk.30.attn_norm.weight to cuda:0
loading blk.30.ffn_norm.weight to cuda:0
loading blk.31.attn_q_a.weight with CPU
loading blk.31.attn_q_a_norm.weight to cuda:0
loading blk.31.attn_q_b.weight with CPU
loading blk.31.attn_kv_a_mqa.weight with CPU
loading blk.31.attn_kv_a_norm.weight to cuda:0
loading blk.31.attn_kv_b.weight to cuda:0
loading blk.31.attn_output.weight with CPU
loading blk.31.ffn_gate_shexp.weight with CPU
loading blk.31.ffn_up_shexp.weight with CPU
loading blk.31.ffn_down_shexp.weight with CPU
loading blk.31.attn_norm.weight to cuda:0
loading blk.31.ffn_norm.weight to cuda:0
loading blk.32.attn_q_a.weight with CPU
loading blk.32.attn_q_a_norm.weight to cuda:0
loading blk.32.attn_q_b.weight with CPU
loading blk.32.attn_kv_a_mqa.weight with CPU
loading blk.32.attn_kv_a_norm.weight to cuda:0
loading blk.32.attn_kv_b.weight to cuda:0
loading blk.32.attn_output.weight with CPU
loading blk.32.ffn_gate_shexp.weight with CPU
loading blk.32.ffn_up_shexp.weight with CPU
loading blk.32.ffn_down_shexp.weight with CPU
loading blk.32.attn_norm.weight to cuda:0
loading blk.32.ffn_norm.weight to cuda:0
loading blk.33.attn_q_a.weight with CPU
loading blk.33.attn_q_a_norm.weight to cuda:0
loading blk.33.attn_q_b.weight with CPU
loading blk.33.attn_kv_a_mqa.weight with CPU
loading blk.33.attn_kv_a_norm.weight to cuda:0
loading blk.33.attn_kv_b.weight to cuda:0
loading blk.33.attn_output.weight with CPU
loading blk.33.ffn_gate_shexp.weight with CPU
loading blk.33.ffn_up_shexp.weight with CPU
loading blk.33.ffn_down_shexp.weight with CPU
loading blk.33.attn_norm.weight to cuda:0
loading blk.33.ffn_norm.weight to cuda:0
loading blk.34.attn_q_a.weight with CPU
loading blk.34.attn_q_a_norm.weight to cuda:0
loading blk.34.attn_q_b.weight with CPU
loading blk.34.attn_kv_a_mqa.weight with CPU
loading blk.34.attn_kv_a_norm.weight to cuda:0
loading blk.34.attn_kv_b.weight to cuda:0
loading blk.34.attn_output.weight with CPU
loading blk.34.ffn_gate_shexp.weight with CPU
loading blk.34.ffn_up_shexp.weight with CPU
loading blk.34.ffn_down_shexp.weight with CPU
loading blk.34.attn_norm.weight to cuda:0
loading blk.34.ffn_norm.weight to cuda:0
loading blk.35.attn_q_a.weight with CPU
loading blk.35.attn_q_a_norm.weight to cuda:0
loading blk.35.attn_q_b.weight with CPU
loading blk.35.attn_kv_a_mqa.weight with CPU
loading blk.35.attn_kv_a_norm.weight to cuda:0
loading blk.35.attn_kv_b.weight to cuda:0
loading blk.35.attn_output.weight with CPU
loading blk.35.ffn_gate_shexp.weight with CPU
loading blk.35.ffn_up_shexp.weight with CPU
loading blk.35.ffn_down_shexp.weight with CPU
loading blk.35.attn_norm.weight to cuda:0
loading blk.35.ffn_norm.weight to cuda:0
loading blk.36.attn_q_a.weight with CPU
loading blk.36.attn_q_a_norm.weight to cuda:0
loading blk.36.attn_q_b.weight with CPU
loading blk.36.attn_kv_a_mqa.weight with CPU
loading blk.36.attn_kv_a_norm.weight to cuda:0
loading blk.36.attn_kv_b.weight to cuda:0
loading blk.36.attn_output.weight with CPU
loading blk.36.ffn_gate_shexp.weight with CPU
loading blk.36.ffn_up_shexp.weight with CPU
loading blk.36.ffn_down_shexp.weight with CPU
loading blk.36.attn_norm.weight to cuda:0
loading blk.36.ffn_norm.weight to cuda:0
loading blk.37.attn_q_a.weight with CPU
loading blk.37.attn_q_a_norm.weight to cuda:0
loading blk.37.attn_q_b.weight with CPU
loading blk.37.attn_kv_a_mqa.weight with CPU
loading blk.37.attn_kv_a_norm.weight to cuda:0
loading blk.37.attn_kv_b.weight to cuda:0
loading blk.37.attn_output.weight with CPU
loading blk.37.ffn_gate_shexp.weight with CPU
loading blk.37.ffn_up_shexp.weight with CPU
loading blk.37.ffn_down_shexp.weight with CPU
loading blk.37.attn_norm.weight to cuda:0
loading blk.37.ffn_norm.weight to cuda:0
loading blk.38.attn_q_a.weight with CPU
loading blk.38.attn_q_a_norm.weight to cuda:0
loading blk.38.attn_q_b.weight with CPU
loading blk.38.attn_kv_a_mqa.weight with CPU
loading blk.38.attn_kv_a_norm.weight to cuda:0
loading blk.38.attn_kv_b.weight to cuda:0
loading blk.38.attn_output.weight with CPU
loading blk.38.ffn_gate_shexp.weight with CPU
loading blk.38.ffn_up_shexp.weight with CPU
loading blk.38.ffn_down_shexp.weight with CPU
loading blk.38.attn_norm.weight to cuda:0
loading blk.38.ffn_norm.weight to cuda:0
loading blk.39.attn_q_a.weight with CPU
loading blk.39.attn_q_a_norm.weight to cuda:0
loading blk.39.attn_q_b.weight with CPU
loading blk.39.attn_kv_a_mqa.weight with CPU
loading blk.39.attn_kv_a_norm.weight to cuda:0
loading blk.39.attn_kv_b.weight to cuda:0
loading blk.39.attn_output.weight with CPU
loading blk.39.ffn_gate_shexp.weight with CPU
loading blk.39.ffn_up_shexp.weight with CPU
loading blk.39.ffn_down_shexp.weight with CPU
loading blk.39.attn_norm.weight to cuda:0
loading blk.39.ffn_norm.weight to cuda:0
loading blk.40.attn_q_a.weight with CPU
loading blk.40.attn_q_a_norm.weight to cuda:0
loading blk.40.attn_q_b.weight with CPU
loading blk.40.attn_kv_a_mqa.weight with CPU
loading blk.40.attn_kv_a_norm.weight to cuda:0
loading blk.40.attn_kv_b.weight to cuda:0
loading blk.40.attn_output.weight with CPU
loading blk.40.ffn_gate_shexp.weight with CPU
loading blk.40.ffn_up_shexp.weight with CPU
loading blk.40.ffn_down_shexp.weight with CPU
loading blk.40.attn_norm.weight to cuda:0
loading blk.40.ffn_norm.weight to cuda:0
loading blk.41.attn_q_a.weight with CPU
loading blk.41.attn_q_a_norm.weight to cuda:0
loading blk.41.attn_q_b.weight with CPU
loading blk.41.attn_kv_a_mqa.weight with CPU
loading blk.41.attn_kv_a_norm.weight to cuda:0
loading blk.41.attn_kv_b.weight to cuda:0
loading blk.41.attn_output.weight with CPU
loading blk.41.ffn_gate_shexp.weight with CPU
loading blk.41.ffn_up_shexp.weight with CPU
loading blk.41.ffn_down_shexp.weight with CPU
loading blk.41.attn_norm.weight to cuda:0
loading blk.41.ffn_norm.weight to cuda:0
loading blk.42.attn_q_a.weight with CPU
loading blk.42.attn_q_a_norm.weight to cuda:0
loading blk.42.attn_q_b.weight with CPU
loading blk.42.attn_kv_a_mqa.weight with CPU
loading blk.42.attn_kv_a_norm.weight to cuda:0
loading blk.42.attn_kv_b.weight to cuda:0
loading blk.42.attn_output.weight with CPU
loading blk.42.ffn_gate_shexp.weight with CPU
loading blk.42.ffn_up_shexp.weight with CPU
loading blk.42.ffn_down_shexp.weight with CPU
loading blk.42.attn_norm.weight to cuda:0
loading blk.42.ffn_norm.weight to cuda:0
loading blk.43.attn_q_a.weight with CPU
loading blk.43.attn_q_a_norm.weight to cuda:0
loading blk.43.attn_q_b.weight with CPU
loading blk.43.attn_kv_a_mqa.weight with CPU
loading blk.43.attn_kv_a_norm.weight to cuda:0
loading blk.43.attn_kv_b.weight to cuda:0
loading blk.43.attn_output.weight with CPU
loading blk.43.ffn_gate_shexp.weight with CPU
loading blk.43.ffn_up_shexp.weight with CPU
loading blk.43.ffn_down_shexp.weight with CPU
loading blk.43.attn_norm.weight to cuda:0
loading blk.43.ffn_norm.weight to cuda:0
loading blk.44.attn_q_a.weight with CPU
loading blk.44.attn_q_a_norm.weight to cuda:0
loading blk.44.attn_q_b.weight with CPU
loading blk.44.attn_kv_a_mqa.weight with CPU
loading blk.44.attn_kv_a_norm.weight to cuda:0
loading blk.44.attn_kv_b.weight to cuda:0
loading blk.44.attn_output.weight with CPU
loading blk.44.ffn_gate_shexp.weight with CPU
loading blk.44.ffn_up_shexp.weight with CPU
loading blk.44.ffn_down_shexp.weight with CPU
loading blk.44.attn_norm.weight to cuda:0
loading blk.44.ffn_norm.weight to cuda:0
loading blk.45.attn_q_a.weight with CPU
loading blk.45.attn_q_a_norm.weight to cuda:0
loading blk.45.attn_q_b.weight with CPU
loading blk.45.attn_kv_a_mqa.weight with CPU
loading blk.45.attn_kv_a_norm.weight to cuda:0
loading blk.45.attn_kv_b.weight to cuda:0
loading blk.45.attn_output.weight with CPU
loading blk.45.ffn_gate_shexp.weight with CPU
loading blk.45.ffn_up_shexp.weight with CPU
loading blk.45.ffn_down_shexp.weight with CPU
loading blk.45.attn_norm.weight to cuda:0
loading blk.45.ffn_norm.weight to cuda:0
loading blk.46.attn_q_a.weight with CPU
loading blk.46.attn_q_a_norm.weight to cuda:0
loading blk.46.attn_q_b.weight with CPU
loading blk.46.attn_kv_a_mqa.weight with CPU
loading blk.46.attn_kv_a_norm.weight to cuda:0
loading blk.46.attn_kv_b.weight to cuda:0
loading blk.46.attn_output.weight with CPU
loading blk.46.ffn_gate_shexp.weight with CPU
loading blk.46.ffn_up_shexp.weight with CPU
loading blk.46.ffn_down_shexp.weight with CPU
loading blk.46.attn_norm.weight to cuda:0
loading blk.46.ffn_norm.weight to cuda:0
loading blk.47.attn_q_a.weight with CPU
loading blk.47.attn_q_a_norm.weight to cuda:0
loading blk.47.attn_q_b.weight with CPU
loading blk.47.attn_kv_a_mqa.weight with CPU
loading blk.47.attn_kv_a_norm.weight to cuda:0
loading blk.47.attn_kv_b.weight to cuda:0
loading blk.47.attn_output.weight with CPU
loading blk.47.ffn_gate_shexp.weight with CPU
loading blk.47.ffn_up_shexp.weight with CPU
loading blk.47.ffn_down_shexp.weight with CPU
loading blk.47.attn_norm.weight to cuda:0
loading blk.47.ffn_norm.weight to cuda:0
loading blk.48.attn_q_a.weight with CPU
loading blk.48.attn_q_a_norm.weight to cuda:0
loading blk.48.attn_q_b.weight with CPU
loading blk.48.attn_kv_a_mqa.weight with CPU
loading blk.48.attn_kv_a_norm.weight to cuda:0
loading blk.48.attn_kv_b.weight to cuda:0
loading blk.48.attn_output.weight with CPU
loading blk.48.ffn_gate_shexp.weight with CPU
loading blk.48.ffn_up_shexp.weight with CPU
loading blk.48.ffn_down_shexp.weight with CPU
loading blk.48.attn_norm.weight to cuda:0
loading blk.48.ffn_norm.weight to cuda:0
loading blk.49.attn_q_a.weight with CPU
loading blk.49.attn_q_a_norm.weight to cuda:0
loading blk.49.attn_q_b.weight with CPU
loading blk.49.attn_kv_a_mqa.weight with CPU
loading blk.49.attn_kv_a_norm.weight to cuda:0
loading blk.49.attn_kv_b.weight to cuda:0
loading blk.49.attn_output.weight with CPU
loading blk.49.ffn_gate_shexp.weight with CPU
loading blk.49.ffn_up_shexp.weight with CPU
loading blk.49.ffn_down_shexp.weight with CPU
loading blk.49.attn_norm.weight to cuda:0
loading blk.49.ffn_norm.weight to cuda:0
loading blk.50.attn_q_a.weight with CPU
loading blk.50.attn_q_a_norm.weight to cuda:0
loading blk.50.attn_q_b.weight with CPU
loading blk.50.attn_kv_a_mqa.weight with CPU
loading blk.50.attn_kv_a_norm.weight to cuda:0
loading blk.50.attn_kv_b.weight to cuda:0
loading blk.50.attn_output.weight with CPU
loading blk.50.ffn_gate_shexp.weight with CPU
loading blk.50.ffn_up_shexp.weight with CPU
loading blk.50.ffn_down_shexp.weight with CPU
loading blk.50.attn_norm.weight to cuda:0
loading blk.50.ffn_norm.weight to cuda:0
loading blk.51.attn_q_a.weight with CPU
loading blk.51.attn_q_a_norm.weight to cuda:0
loading blk.51.attn_q_b.weight with CPU
loading blk.51.attn_kv_a_mqa.weight with CPU
loading blk.51.attn_kv_a_norm.weight to cuda:0
loading blk.51.attn_kv_b.weight to cuda:0
loading blk.51.attn_output.weight with CPU
loading blk.51.ffn_gate_shexp.weight with CPU
loading blk.51.ffn_up_shexp.weight with CPU
loading blk.51.ffn_down_shexp.weight with CPU
loading blk.51.attn_norm.weight to cuda:0
loading blk.51.ffn_norm.weight to cuda:0
loading blk.52.attn_q_a.weight with CPU
loading blk.52.attn_q_a_norm.weight to cuda:0
loading blk.52.attn_q_b.weight with CPU
loading blk.52.attn_kv_a_mqa.weight with CPU
loading blk.52.attn_kv_a_norm.weight to cuda:0
loading blk.52.attn_kv_b.weight to cuda:0
loading blk.52.attn_output.weight with CPU
loading blk.52.ffn_gate_shexp.weight with CPU
loading blk.52.ffn_up_shexp.weight with CPU
loading blk.52.ffn_down_shexp.weight with CPU
loading blk.52.attn_norm.weight to cuda:0
loading blk.52.ffn_norm.weight to cuda:0
loading blk.53.attn_q_a.weight with CPU
loading blk.53.attn_q_a_norm.weight to cuda:0
loading blk.53.attn_q_b.weight with CPU
loading blk.53.attn_kv_a_mqa.weight with CPU
loading blk.53.attn_kv_a_norm.weight to cuda:0
loading blk.53.attn_kv_b.weight to cuda:0
loading blk.53.attn_output.weight with CPU
loading blk.53.ffn_gate_shexp.weight with CPU
loading blk.53.ffn_up_shexp.weight with CPU
loading blk.53.ffn_down_shexp.weight with CPU
loading blk.53.attn_norm.weight to cuda:0
loading blk.53.ffn_norm.weight to cuda:0
loading blk.54.attn_q_a.weight with CPU
loading blk.54.attn_q_a_norm.weight to cuda:0
loading blk.54.attn_q_b.weight with CPU
loading blk.54.attn_kv_a_mqa.weight with CPU
loading blk.54.attn_kv_a_norm.weight to cuda:0
loading blk.54.attn_kv_b.weight to cuda:0
loading blk.54.attn_output.weight with CPU
loading blk.54.ffn_gate_shexp.weight with CPU
loading blk.54.ffn_up_shexp.weight with CPU
loading blk.54.ffn_down_shexp.weight with CPU
loading blk.54.attn_norm.weight to cuda:0
loading blk.54.ffn_norm.weight to cuda:0
loading blk.55.attn_q_a.weight with CPU
loading blk.55.attn_q_a_norm.weight to cuda:0
loading blk.55.attn_q_b.weight with CPU
loading blk.55.attn_kv_a_mqa.weight with CPU
loading blk.55.attn_kv_a_norm.weight to cuda:0
loading blk.55.attn_kv_b.weight to cuda:0
loading blk.55.attn_output.weight with CPU
loading blk.55.ffn_gate_shexp.weight with CPU
loading blk.55.ffn_up_shexp.weight with CPU
loading blk.55.ffn_down_shexp.weight with CPU
loading blk.55.attn_norm.weight to cuda:0
loading blk.55.ffn_norm.weight to cuda:0
loading blk.56.attn_q_a.weight with CPU
loading blk.56.attn_q_a_norm.weight to cuda:0
loading blk.56.attn_q_b.weight with CPU
loading blk.56.attn_kv_a_mqa.weight with CPU
loading blk.56.attn_kv_a_norm.weight to cuda:0
loading blk.56.attn_kv_b.weight to cuda:0
loading blk.56.attn_output.weight with CPU
loading blk.56.ffn_gate_shexp.weight with CPU
loading blk.56.ffn_up_shexp.weight with CPU
loading blk.56.ffn_down_shexp.weight with CPU
loading blk.56.attn_norm.weight to cuda:0
loading blk.56.ffn_norm.weight to cuda:0
loading blk.57.attn_q_a.weight with CPU
loading blk.57.attn_q_a_norm.weight to cuda:0
loading blk.57.attn_q_b.weight with CPU
loading blk.57.attn_kv_a_mqa.weight with CPU
loading blk.57.attn_kv_a_norm.weight to cuda:0
loading blk.57.attn_kv_b.weight to cuda:0
loading blk.57.attn_output.weight with CPU
loading blk.57.ffn_gate_shexp.weight with CPU
loading blk.57.ffn_up_shexp.weight with CPU
loading blk.57.ffn_down_shexp.weight with CPU
loading blk.57.attn_norm.weight to cuda:0
loading blk.57.ffn_norm.weight to cuda:0
loading blk.58.attn_q_a.weight with CPU
loading blk.58.attn_q_a_norm.weight to cuda:0
loading blk.58.attn_q_b.weight with CPU
loading blk.58.attn_kv_a_mqa.weight with CPU
loading blk.58.attn_kv_a_norm.weight to cuda:0
loading blk.58.attn_kv_b.weight to cuda:0
loading blk.58.attn_output.weight with CPU
loading blk.58.ffn_gate_shexp.weight with CPU
loading blk.58.ffn_up_shexp.weight with CPU
loading blk.58.ffn_down_shexp.weight with CPU
loading blk.58.attn_norm.weight to cuda:0
loading blk.58.ffn_norm.weight to cuda:0
loading blk.59.attn_q_a.weight with CPU
loading blk.59.attn_q_a_norm.weight to cuda:0
loading blk.59.attn_q_b.weight with CPU
loading blk.59.attn_kv_a_mqa.weight with CPU
loading blk.59.attn_kv_a_norm.weight to cuda:0
loading blk.59.attn_kv_b.weight to cuda:0
loading blk.59.attn_output.weight with CPU
loading blk.59.ffn_gate_shexp.weight with CPU
loading blk.59.ffn_up_shexp.weight with CPU
loading blk.59.ffn_down_shexp.weight with CPU
loading blk.59.attn_norm.weight to cuda:0
loading blk.59.ffn_norm.weight to cuda:0
loading blk.60.attn_q_a.weight with CPU
loading blk.60.attn_q_a_norm.weight to cuda:0
loading blk.60.attn_q_b.weight with CPU
loading blk.60.attn_kv_a_mqa.weight with CPU
loading blk.60.attn_kv_a_norm.weight to cuda:0
loading blk.60.attn_kv_b.weight to cuda:0
loading blk.60.attn_output.weight with CPU
loading blk.60.ffn_gate_shexp.weight with CPU
loading blk.60.ffn_up_shexp.weight with CPU
loading blk.60.ffn_down_shexp.weight with CPU
loading blk.60.attn_norm.weight to cuda:0
loading blk.60.ffn_norm.weight to cuda:0
loading output_norm.weight to cuda:0
loading output to cuda using CPUInfer
Chat: hello
/home/ubuntu/repos/kvcache-ai/ktransformers/ktransformers/operators/RoPE.py:269: UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:310.)
  freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
malloc(): unaligned tcache chunk detected
Aborted (core dumped)

and somtimes instead of malloc error, it show

Segmentation fault (core dumped)

Reproduction

/home/ubuntu/pythons/python-3.12.9/bin/python3.12 -m ktransformers.local_chat --model_path /home/ubuntu/resources/models/deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntu/resources/models/unsloth/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf --optimize_config_path /home/ubuntu/repos/kvcache-ai/ktransformers/ktransformers/optimize/optimize_rules/rocm/DeepSeek-V3-Chat.yaml --cpu_infer 28

Environment

- info
OS Ubuntu24.04
CPU AMD 7F52
GPU AMD 7900XTX
RAM 256G 2666
Originally created by @Ryuukeisyou on GitHub (Mar 16, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/909 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug Hello and thank you for your work! I built ktransformers and tried server times to run DeepSeek-R1-UD-IQ1_S, there is only one time it generated several tokens (very slow, much slower than using llma.cpp) and then crash, and most of the time it crashed without generating anything. Output: ``` loading output to cuda using CPUInfer loading token_embd.weight to cpu loading token_embd.weight with CPU loading blk.0.attn_q_a.weight with CPU loading blk.0.attn_q_a_norm.weight to cuda:0 loading blk.0.attn_q_b.weight with CPU loading blk.0.attn_kv_a_mqa.weight with CPU loading blk.0.attn_kv_a_norm.weight to cuda:0 loading blk.0.attn_kv_b.weight to cuda:0 loading blk.0.attn_output.weight with CPU loading blk.0.ffn_gate.weight with CPU loading blk.0.ffn_up.weight with CPU loading blk.0.ffn_down.weight with CPU loading blk.0.attn_norm.weight to cuda:0 loading blk.0.ffn_norm.weight to cuda:0 loading blk.1.attn_q_a.weight with CPU loading blk.1.attn_q_a_norm.weight to cuda:0 loading blk.1.attn_q_b.weight with CPU loading blk.1.attn_kv_a_mqa.weight with CPU loading blk.1.attn_kv_a_norm.weight to cuda:0 loading blk.1.attn_kv_b.weight to cuda:0 loading blk.1.attn_output.weight with CPU loading blk.1.ffn_gate.weight with CPU loading blk.1.ffn_up.weight with CPU loading blk.1.ffn_down.weight with CPU loading blk.1.attn_norm.weight to cuda:0 loading blk.1.ffn_norm.weight to cuda:0 loading blk.2.attn_q_a.weight with CPU loading blk.2.attn_q_a_norm.weight to cuda:0 loading blk.2.attn_q_b.weight with CPU loading blk.2.attn_kv_a_mqa.weight with CPU loading blk.2.attn_kv_a_norm.weight to cuda:0 loading blk.2.attn_kv_b.weight to cuda:0 loading blk.2.attn_output.weight with CPU loading blk.2.ffn_gate.weight with CPU loading blk.2.ffn_up.weight with CPU loading blk.2.ffn_down.weight with CPU loading blk.2.attn_norm.weight to cuda:0 loading blk.2.ffn_norm.weight to cuda:0 loading blk.3.attn_q_a.weight with CPU loading blk.3.attn_q_a_norm.weight to cuda:0 loading blk.3.attn_q_b.weight with CPU loading blk.3.attn_kv_a_mqa.weight with CPU loading blk.3.attn_kv_a_norm.weight to cuda:0 loading blk.3.attn_kv_b.weight to cuda:0 loading blk.3.attn_output.weight with CPU loading blk.3.ffn_gate_shexp.weight with CPU loading blk.3.ffn_up_shexp.weight with CPU loading blk.3.ffn_down_shexp.weight with CPU loading blk.3.attn_norm.weight to cuda:0 loading blk.3.ffn_norm.weight to cuda:0 loading blk.4.attn_q_a.weight with CPU loading blk.4.attn_q_a_norm.weight to cuda:0 loading blk.4.attn_q_b.weight with CPU loading blk.4.attn_kv_a_mqa.weight with CPU loading blk.4.attn_kv_a_norm.weight to cuda:0 loading blk.4.attn_kv_b.weight to cuda:0 loading blk.4.attn_output.weight with CPU loading blk.4.ffn_gate_shexp.weight with CPU loading blk.4.ffn_up_shexp.weight with CPU loading blk.4.ffn_down_shexp.weight with CPU loading blk.4.attn_norm.weight to cuda:0 loading blk.4.ffn_norm.weight to cuda:0 loading blk.5.attn_q_a.weight with CPU loading blk.5.attn_q_a_norm.weight to cuda:0 loading blk.5.attn_q_b.weight with CPU loading blk.5.attn_kv_a_mqa.weight with CPU loading blk.5.attn_kv_a_norm.weight to cuda:0 loading blk.5.attn_kv_b.weight to cuda:0 loading blk.5.attn_output.weight with CPU loading blk.5.ffn_gate_shexp.weight with CPU loading blk.5.ffn_up_shexp.weight with CPU loading blk.5.ffn_down_shexp.weight with CPU loading blk.5.attn_norm.weight to cuda:0 loading blk.5.ffn_norm.weight to cuda:0 loading blk.6.attn_q_a.weight with CPU loading blk.6.attn_q_a_norm.weight to cuda:0 loading blk.6.attn_q_b.weight with CPU loading blk.6.attn_kv_a_mqa.weight with CPU loading blk.6.attn_kv_a_norm.weight to cuda:0 loading blk.6.attn_kv_b.weight to cuda:0 loading blk.6.attn_output.weight with CPU loading blk.6.ffn_gate_shexp.weight with CPU loading blk.6.ffn_up_shexp.weight with CPU loading blk.6.ffn_down_shexp.weight with CPU loading blk.6.attn_norm.weight to cuda:0 loading blk.6.ffn_norm.weight to cuda:0 loading blk.7.attn_q_a.weight with CPU loading blk.7.attn_q_a_norm.weight to cuda:0 loading blk.7.attn_q_b.weight with CPU loading blk.7.attn_kv_a_mqa.weight with CPU loading blk.7.attn_kv_a_norm.weight to cuda:0 loading blk.7.attn_kv_b.weight to cuda:0 loading blk.7.attn_output.weight with CPU loading blk.7.ffn_gate_shexp.weight with CPU loading blk.7.ffn_up_shexp.weight with CPU loading blk.7.ffn_down_shexp.weight with CPU loading blk.7.attn_norm.weight to cuda:0 loading blk.7.ffn_norm.weight to cuda:0 loading blk.8.attn_q_a.weight with CPU loading blk.8.attn_q_a_norm.weight to cuda:0 loading blk.8.attn_q_b.weight with CPU loading blk.8.attn_kv_a_mqa.weight with CPU loading blk.8.attn_kv_a_norm.weight to cuda:0 loading blk.8.attn_kv_b.weight to cuda:0 loading blk.8.attn_output.weight with CPU loading blk.8.ffn_gate_shexp.weight with CPU loading blk.8.ffn_up_shexp.weight with CPU loading blk.8.ffn_down_shexp.weight with CPU loading blk.8.attn_norm.weight to cuda:0 loading blk.8.ffn_norm.weight to cuda:0 loading blk.9.attn_q_a.weight with CPU loading blk.9.attn_q_a_norm.weight to cuda:0 loading blk.9.attn_q_b.weight with CPU loading blk.9.attn_kv_a_mqa.weight with CPU loading blk.9.attn_kv_a_norm.weight to cuda:0 loading blk.9.attn_kv_b.weight to cuda:0 loading blk.9.attn_output.weight with CPU loading blk.9.ffn_gate_shexp.weight with CPU loading blk.9.ffn_up_shexp.weight with CPU loading blk.9.ffn_down_shexp.weight with CPU loading blk.9.attn_norm.weight to cuda:0 loading blk.9.ffn_norm.weight to cuda:0 loading blk.10.attn_q_a.weight with CPU loading blk.10.attn_q_a_norm.weight to cuda:0 loading blk.10.attn_q_b.weight with CPU loading blk.10.attn_kv_a_mqa.weight with CPU loading blk.10.attn_kv_a_norm.weight to cuda:0 loading blk.10.attn_kv_b.weight to cuda:0 loading blk.10.attn_output.weight with CPU loading blk.10.ffn_gate_shexp.weight with CPU loading blk.10.ffn_up_shexp.weight with CPU loading blk.10.ffn_down_shexp.weight with CPU loading blk.10.attn_norm.weight to cuda:0 loading blk.10.ffn_norm.weight to cuda:0 loading blk.11.attn_q_a.weight with CPU loading blk.11.attn_q_a_norm.weight to cuda:0 loading blk.11.attn_q_b.weight with CPU loading blk.11.attn_kv_a_mqa.weight with CPU loading blk.11.attn_kv_a_norm.weight to cuda:0 loading blk.11.attn_kv_b.weight to cuda:0 loading blk.11.attn_output.weight with CPU loading blk.11.ffn_gate_shexp.weight with CPU loading blk.11.ffn_up_shexp.weight with CPU loading blk.11.ffn_down_shexp.weight with CPU loading blk.11.attn_norm.weight to cuda:0 loading blk.11.ffn_norm.weight to cuda:0 loading blk.12.attn_q_a.weight with CPU loading blk.12.attn_q_a_norm.weight to cuda:0 loading blk.12.attn_q_b.weight with CPU loading blk.12.attn_kv_a_mqa.weight with CPU loading blk.12.attn_kv_a_norm.weight to cuda:0 loading blk.12.attn_kv_b.weight to cuda:0 loading blk.12.attn_output.weight with CPU loading blk.12.ffn_gate_shexp.weight with CPU loading blk.12.ffn_up_shexp.weight with CPU loading blk.12.ffn_down_shexp.weight with CPU loading blk.12.attn_norm.weight to cuda:0 loading blk.12.ffn_norm.weight to cuda:0 loading blk.13.attn_q_a.weight with CPU loading blk.13.attn_q_a_norm.weight to cuda:0 loading blk.13.attn_q_b.weight with CPU loading blk.13.attn_kv_a_mqa.weight with CPU loading blk.13.attn_kv_a_norm.weight to cuda:0 loading blk.13.attn_kv_b.weight to cuda:0 loading blk.13.attn_output.weight with CPU loading blk.13.ffn_gate_shexp.weight with CPU loading blk.13.ffn_up_shexp.weight with CPU loading blk.13.ffn_down_shexp.weight with CPU loading blk.13.attn_norm.weight to cuda:0 loading blk.13.ffn_norm.weight to cuda:0 loading blk.14.attn_q_a.weight with CPU loading blk.14.attn_q_a_norm.weight to cuda:0 loading blk.14.attn_q_b.weight with CPU loading blk.14.attn_kv_a_mqa.weight with CPU loading blk.14.attn_kv_a_norm.weight to cuda:0 loading blk.14.attn_kv_b.weight to cuda:0 loading blk.14.attn_output.weight with CPU loading blk.14.ffn_gate_shexp.weight with CPU loading blk.14.ffn_up_shexp.weight with CPU loading blk.14.ffn_down_shexp.weight with CPU loading blk.14.attn_norm.weight to cuda:0 loading blk.14.ffn_norm.weight to cuda:0 loading blk.15.attn_q_a.weight with CPU loading blk.15.attn_q_a_norm.weight to cuda:0 loading blk.15.attn_q_b.weight with CPU loading blk.15.attn_kv_a_mqa.weight with CPU loading blk.15.attn_kv_a_norm.weight to cuda:0 loading blk.15.attn_kv_b.weight to cuda:0 loading blk.15.attn_output.weight with CPU loading blk.15.ffn_gate_shexp.weight with CPU loading blk.15.ffn_up_shexp.weight with CPU loading blk.15.ffn_down_shexp.weight with CPU loading blk.15.attn_norm.weight to cuda:0 loading blk.15.ffn_norm.weight to cuda:0 loading blk.16.attn_q_a.weight with CPU loading blk.16.attn_q_a_norm.weight to cuda:0 loading blk.16.attn_q_b.weight with CPU loading blk.16.attn_kv_a_mqa.weight with CPU loading blk.16.attn_kv_a_norm.weight to cuda:0 loading blk.16.attn_kv_b.weight to cuda:0 loading blk.16.attn_output.weight with CPU loading blk.16.ffn_gate_shexp.weight with CPU loading blk.16.ffn_up_shexp.weight with CPU loading blk.16.ffn_down_shexp.weight with CPU loading blk.16.attn_norm.weight to cuda:0 loading blk.16.ffn_norm.weight to cuda:0 loading blk.17.attn_q_a.weight with CPU loading blk.17.attn_q_a_norm.weight to cuda:0 loading blk.17.attn_q_b.weight with CPU loading blk.17.attn_kv_a_mqa.weight with CPU loading blk.17.attn_kv_a_norm.weight to cuda:0 loading blk.17.attn_kv_b.weight to cuda:0 loading blk.17.attn_output.weight with CPU loading blk.17.ffn_gate_shexp.weight with CPU loading blk.17.ffn_up_shexp.weight with CPU loading blk.17.ffn_down_shexp.weight with CPU loading blk.17.attn_norm.weight to cuda:0 loading blk.17.ffn_norm.weight to cuda:0 loading blk.18.attn_q_a.weight with CPU loading blk.18.attn_q_a_norm.weight to cuda:0 loading blk.18.attn_q_b.weight with CPU loading blk.18.attn_kv_a_mqa.weight with CPU loading blk.18.attn_kv_a_norm.weight to cuda:0 loading blk.18.attn_kv_b.weight to cuda:0 loading blk.18.attn_output.weight with CPU loading blk.18.ffn_gate_shexp.weight with CPU loading blk.18.ffn_up_shexp.weight with CPU loading blk.18.ffn_down_shexp.weight with CPU loading blk.18.attn_norm.weight to cuda:0 loading blk.18.ffn_norm.weight to cuda:0 loading blk.19.attn_q_a.weight with CPU loading blk.19.attn_q_a_norm.weight to cuda:0 loading blk.19.attn_q_b.weight with CPU loading blk.19.attn_kv_a_mqa.weight with CPU loading blk.19.attn_kv_a_norm.weight to cuda:0 loading blk.19.attn_kv_b.weight to cuda:0 loading blk.19.attn_output.weight with CPU loading blk.19.ffn_gate_shexp.weight with CPU loading blk.19.ffn_up_shexp.weight with CPU loading blk.19.ffn_down_shexp.weight with CPU loading blk.19.attn_norm.weight to cuda:0 loading blk.19.ffn_norm.weight to cuda:0 loading blk.20.attn_q_a.weight with CPU loading blk.20.attn_q_a_norm.weight to cuda:0 loading blk.20.attn_q_b.weight with CPU loading blk.20.attn_kv_a_mqa.weight with CPU loading blk.20.attn_kv_a_norm.weight to cuda:0 loading blk.20.attn_kv_b.weight to cuda:0 loading blk.20.attn_output.weight with CPU loading blk.20.ffn_gate_shexp.weight with CPU loading blk.20.ffn_up_shexp.weight with CPU loading blk.20.ffn_down_shexp.weight with CPU loading blk.20.attn_norm.weight to cuda:0 loading blk.20.ffn_norm.weight to cuda:0 loading blk.21.attn_q_a.weight with CPU loading blk.21.attn_q_a_norm.weight to cuda:0 loading blk.21.attn_q_b.weight with CPU loading blk.21.attn_kv_a_mqa.weight with CPU loading blk.21.attn_kv_a_norm.weight to cuda:0 loading blk.21.attn_kv_b.weight to cuda:0 loading blk.21.attn_output.weight with CPU loading blk.21.ffn_gate_shexp.weight with CPU loading blk.21.ffn_up_shexp.weight with CPU loading blk.21.ffn_down_shexp.weight with CPU loading blk.21.attn_norm.weight to cuda:0 loading blk.21.ffn_norm.weight to cuda:0 loading blk.22.attn_q_a.weight with CPU loading blk.22.attn_q_a_norm.weight to cuda:0 loading blk.22.attn_q_b.weight with CPU loading blk.22.attn_kv_a_mqa.weight with CPU loading blk.22.attn_kv_a_norm.weight to cuda:0 loading blk.22.attn_kv_b.weight to cuda:0 loading blk.22.attn_output.weight with CPU loading blk.22.ffn_gate_shexp.weight with CPU loading blk.22.ffn_up_shexp.weight with CPU loading blk.22.ffn_down_shexp.weight with CPU loading blk.22.attn_norm.weight to cuda:0 loading blk.22.ffn_norm.weight to cuda:0 loading blk.23.attn_q_a.weight with CPU loading blk.23.attn_q_a_norm.weight to cuda:0 loading blk.23.attn_q_b.weight with CPU loading blk.23.attn_kv_a_mqa.weight with CPU loading blk.23.attn_kv_a_norm.weight to cuda:0 loading blk.23.attn_kv_b.weight to cuda:0 loading blk.23.attn_output.weight with CPU loading blk.23.ffn_gate_shexp.weight with CPU loading blk.23.ffn_up_shexp.weight with CPU loading blk.23.ffn_down_shexp.weight with CPU loading blk.23.attn_norm.weight to cuda:0 loading blk.23.ffn_norm.weight to cuda:0 loading blk.24.attn_q_a.weight with CPU loading blk.24.attn_q_a_norm.weight to cuda:0 loading blk.24.attn_q_b.weight with CPU loading blk.24.attn_kv_a_mqa.weight with CPU loading blk.24.attn_kv_a_norm.weight to cuda:0 loading blk.24.attn_kv_b.weight to cuda:0 loading blk.24.attn_output.weight with CPU loading blk.24.ffn_gate_shexp.weight with CPU loading blk.24.ffn_up_shexp.weight with CPU loading blk.24.ffn_down_shexp.weight with CPU loading blk.24.attn_norm.weight to cuda:0 loading blk.24.ffn_norm.weight to cuda:0 loading blk.25.attn_q_a.weight with CPU loading blk.25.attn_q_a_norm.weight to cuda:0 loading blk.25.attn_q_b.weight with CPU loading blk.25.attn_kv_a_mqa.weight with CPU loading blk.25.attn_kv_a_norm.weight to cuda:0 loading blk.25.attn_kv_b.weight to cuda:0 loading blk.25.attn_output.weight with CPU loading blk.25.ffn_gate_shexp.weight with CPU loading blk.25.ffn_up_shexp.weight with CPU loading blk.25.ffn_down_shexp.weight with CPU loading blk.25.attn_norm.weight to cuda:0 loading blk.25.ffn_norm.weight to cuda:0 loading blk.26.attn_q_a.weight with CPU loading blk.26.attn_q_a_norm.weight to cuda:0 loading blk.26.attn_q_b.weight with CPU loading blk.26.attn_kv_a_mqa.weight with CPU loading blk.26.attn_kv_a_norm.weight to cuda:0 loading blk.26.attn_kv_b.weight to cuda:0 loading blk.26.attn_output.weight with CPU loading blk.26.ffn_gate_shexp.weight with CPU loading blk.26.ffn_up_shexp.weight with CPU loading blk.26.ffn_down_shexp.weight with CPU loading blk.26.attn_norm.weight to cuda:0 loading blk.26.ffn_norm.weight to cuda:0 loading blk.27.attn_q_a.weight with CPU loading blk.27.attn_q_a_norm.weight to cuda:0 loading blk.27.attn_q_b.weight with CPU loading blk.27.attn_kv_a_mqa.weight with CPU loading blk.27.attn_kv_a_norm.weight to cuda:0 loading blk.27.attn_kv_b.weight to cuda:0 loading blk.27.attn_output.weight with CPU loading blk.27.ffn_gate_shexp.weight with CPU loading blk.27.ffn_up_shexp.weight with CPU loading blk.27.ffn_down_shexp.weight with CPU loading blk.27.attn_norm.weight to cuda:0 loading blk.27.ffn_norm.weight to cuda:0 loading blk.28.attn_q_a.weight with CPU loading blk.28.attn_q_a_norm.weight to cuda:0 loading blk.28.attn_q_b.weight with CPU loading blk.28.attn_kv_a_mqa.weight with CPU loading blk.28.attn_kv_a_norm.weight to cuda:0 loading blk.28.attn_kv_b.weight to cuda:0 loading blk.28.attn_output.weight with CPU loading blk.28.ffn_gate_shexp.weight with CPU loading blk.28.ffn_up_shexp.weight with CPU loading blk.28.ffn_down_shexp.weight with CPU loading blk.28.attn_norm.weight to cuda:0 loading blk.28.ffn_norm.weight to cuda:0 loading blk.29.attn_q_a.weight with CPU loading blk.29.attn_q_a_norm.weight to cuda:0 loading blk.29.attn_q_b.weight with CPU loading blk.29.attn_kv_a_mqa.weight with CPU loading blk.29.attn_kv_a_norm.weight to cuda:0 loading blk.29.attn_kv_b.weight to cuda:0 loading blk.29.attn_output.weight with CPU loading blk.29.ffn_gate_shexp.weight with CPU loading blk.29.ffn_up_shexp.weight with CPU loading blk.29.ffn_down_shexp.weight with CPU loading blk.29.attn_norm.weight to cuda:0 loading blk.29.ffn_norm.weight to cuda:0 loading blk.30.attn_q_a.weight with CPU loading blk.30.attn_q_a_norm.weight to cuda:0 loading blk.30.attn_q_b.weight with CPU loading blk.30.attn_kv_a_mqa.weight with CPU loading blk.30.attn_kv_a_norm.weight to cuda:0 loading blk.30.attn_kv_b.weight to cuda:0 loading blk.30.attn_output.weight with CPU loading blk.30.ffn_gate_shexp.weight with CPU loading blk.30.ffn_up_shexp.weight with CPU loading blk.30.ffn_down_shexp.weight with CPU loading blk.30.attn_norm.weight to cuda:0 loading blk.30.ffn_norm.weight to cuda:0 loading blk.31.attn_q_a.weight with CPU loading blk.31.attn_q_a_norm.weight to cuda:0 loading blk.31.attn_q_b.weight with CPU loading blk.31.attn_kv_a_mqa.weight with CPU loading blk.31.attn_kv_a_norm.weight to cuda:0 loading blk.31.attn_kv_b.weight to cuda:0 loading blk.31.attn_output.weight with CPU loading blk.31.ffn_gate_shexp.weight with CPU loading blk.31.ffn_up_shexp.weight with CPU loading blk.31.ffn_down_shexp.weight with CPU loading blk.31.attn_norm.weight to cuda:0 loading blk.31.ffn_norm.weight to cuda:0 loading blk.32.attn_q_a.weight with CPU loading blk.32.attn_q_a_norm.weight to cuda:0 loading blk.32.attn_q_b.weight with CPU loading blk.32.attn_kv_a_mqa.weight with CPU loading blk.32.attn_kv_a_norm.weight to cuda:0 loading blk.32.attn_kv_b.weight to cuda:0 loading blk.32.attn_output.weight with CPU loading blk.32.ffn_gate_shexp.weight with CPU loading blk.32.ffn_up_shexp.weight with CPU loading blk.32.ffn_down_shexp.weight with CPU loading blk.32.attn_norm.weight to cuda:0 loading blk.32.ffn_norm.weight to cuda:0 loading blk.33.attn_q_a.weight with CPU loading blk.33.attn_q_a_norm.weight to cuda:0 loading blk.33.attn_q_b.weight with CPU loading blk.33.attn_kv_a_mqa.weight with CPU loading blk.33.attn_kv_a_norm.weight to cuda:0 loading blk.33.attn_kv_b.weight to cuda:0 loading blk.33.attn_output.weight with CPU loading blk.33.ffn_gate_shexp.weight with CPU loading blk.33.ffn_up_shexp.weight with CPU loading blk.33.ffn_down_shexp.weight with CPU loading blk.33.attn_norm.weight to cuda:0 loading blk.33.ffn_norm.weight to cuda:0 loading blk.34.attn_q_a.weight with CPU loading blk.34.attn_q_a_norm.weight to cuda:0 loading blk.34.attn_q_b.weight with CPU loading blk.34.attn_kv_a_mqa.weight with CPU loading blk.34.attn_kv_a_norm.weight to cuda:0 loading blk.34.attn_kv_b.weight to cuda:0 loading blk.34.attn_output.weight with CPU loading blk.34.ffn_gate_shexp.weight with CPU loading blk.34.ffn_up_shexp.weight with CPU loading blk.34.ffn_down_shexp.weight with CPU loading blk.34.attn_norm.weight to cuda:0 loading blk.34.ffn_norm.weight to cuda:0 loading blk.35.attn_q_a.weight with CPU loading blk.35.attn_q_a_norm.weight to cuda:0 loading blk.35.attn_q_b.weight with CPU loading blk.35.attn_kv_a_mqa.weight with CPU loading blk.35.attn_kv_a_norm.weight to cuda:0 loading blk.35.attn_kv_b.weight to cuda:0 loading blk.35.attn_output.weight with CPU loading blk.35.ffn_gate_shexp.weight with CPU loading blk.35.ffn_up_shexp.weight with CPU loading blk.35.ffn_down_shexp.weight with CPU loading blk.35.attn_norm.weight to cuda:0 loading blk.35.ffn_norm.weight to cuda:0 loading blk.36.attn_q_a.weight with CPU loading blk.36.attn_q_a_norm.weight to cuda:0 loading blk.36.attn_q_b.weight with CPU loading blk.36.attn_kv_a_mqa.weight with CPU loading blk.36.attn_kv_a_norm.weight to cuda:0 loading blk.36.attn_kv_b.weight to cuda:0 loading blk.36.attn_output.weight with CPU loading blk.36.ffn_gate_shexp.weight with CPU loading blk.36.ffn_up_shexp.weight with CPU loading blk.36.ffn_down_shexp.weight with CPU loading blk.36.attn_norm.weight to cuda:0 loading blk.36.ffn_norm.weight to cuda:0 loading blk.37.attn_q_a.weight with CPU loading blk.37.attn_q_a_norm.weight to cuda:0 loading blk.37.attn_q_b.weight with CPU loading blk.37.attn_kv_a_mqa.weight with CPU loading blk.37.attn_kv_a_norm.weight to cuda:0 loading blk.37.attn_kv_b.weight to cuda:0 loading blk.37.attn_output.weight with CPU loading blk.37.ffn_gate_shexp.weight with CPU loading blk.37.ffn_up_shexp.weight with CPU loading blk.37.ffn_down_shexp.weight with CPU loading blk.37.attn_norm.weight to cuda:0 loading blk.37.ffn_norm.weight to cuda:0 loading blk.38.attn_q_a.weight with CPU loading blk.38.attn_q_a_norm.weight to cuda:0 loading blk.38.attn_q_b.weight with CPU loading blk.38.attn_kv_a_mqa.weight with CPU loading blk.38.attn_kv_a_norm.weight to cuda:0 loading blk.38.attn_kv_b.weight to cuda:0 loading blk.38.attn_output.weight with CPU loading blk.38.ffn_gate_shexp.weight with CPU loading blk.38.ffn_up_shexp.weight with CPU loading blk.38.ffn_down_shexp.weight with CPU loading blk.38.attn_norm.weight to cuda:0 loading blk.38.ffn_norm.weight to cuda:0 loading blk.39.attn_q_a.weight with CPU loading blk.39.attn_q_a_norm.weight to cuda:0 loading blk.39.attn_q_b.weight with CPU loading blk.39.attn_kv_a_mqa.weight with CPU loading blk.39.attn_kv_a_norm.weight to cuda:0 loading blk.39.attn_kv_b.weight to cuda:0 loading blk.39.attn_output.weight with CPU loading blk.39.ffn_gate_shexp.weight with CPU loading blk.39.ffn_up_shexp.weight with CPU loading blk.39.ffn_down_shexp.weight with CPU loading blk.39.attn_norm.weight to cuda:0 loading blk.39.ffn_norm.weight to cuda:0 loading blk.40.attn_q_a.weight with CPU loading blk.40.attn_q_a_norm.weight to cuda:0 loading blk.40.attn_q_b.weight with CPU loading blk.40.attn_kv_a_mqa.weight with CPU loading blk.40.attn_kv_a_norm.weight to cuda:0 loading blk.40.attn_kv_b.weight to cuda:0 loading blk.40.attn_output.weight with CPU loading blk.40.ffn_gate_shexp.weight with CPU loading blk.40.ffn_up_shexp.weight with CPU loading blk.40.ffn_down_shexp.weight with CPU loading blk.40.attn_norm.weight to cuda:0 loading blk.40.ffn_norm.weight to cuda:0 loading blk.41.attn_q_a.weight with CPU loading blk.41.attn_q_a_norm.weight to cuda:0 loading blk.41.attn_q_b.weight with CPU loading blk.41.attn_kv_a_mqa.weight with CPU loading blk.41.attn_kv_a_norm.weight to cuda:0 loading blk.41.attn_kv_b.weight to cuda:0 loading blk.41.attn_output.weight with CPU loading blk.41.ffn_gate_shexp.weight with CPU loading blk.41.ffn_up_shexp.weight with CPU loading blk.41.ffn_down_shexp.weight with CPU loading blk.41.attn_norm.weight to cuda:0 loading blk.41.ffn_norm.weight to cuda:0 loading blk.42.attn_q_a.weight with CPU loading blk.42.attn_q_a_norm.weight to cuda:0 loading blk.42.attn_q_b.weight with CPU loading blk.42.attn_kv_a_mqa.weight with CPU loading blk.42.attn_kv_a_norm.weight to cuda:0 loading blk.42.attn_kv_b.weight to cuda:0 loading blk.42.attn_output.weight with CPU loading blk.42.ffn_gate_shexp.weight with CPU loading blk.42.ffn_up_shexp.weight with CPU loading blk.42.ffn_down_shexp.weight with CPU loading blk.42.attn_norm.weight to cuda:0 loading blk.42.ffn_norm.weight to cuda:0 loading blk.43.attn_q_a.weight with CPU loading blk.43.attn_q_a_norm.weight to cuda:0 loading blk.43.attn_q_b.weight with CPU loading blk.43.attn_kv_a_mqa.weight with CPU loading blk.43.attn_kv_a_norm.weight to cuda:0 loading blk.43.attn_kv_b.weight to cuda:0 loading blk.43.attn_output.weight with CPU loading blk.43.ffn_gate_shexp.weight with CPU loading blk.43.ffn_up_shexp.weight with CPU loading blk.43.ffn_down_shexp.weight with CPU loading blk.43.attn_norm.weight to cuda:0 loading blk.43.ffn_norm.weight to cuda:0 loading blk.44.attn_q_a.weight with CPU loading blk.44.attn_q_a_norm.weight to cuda:0 loading blk.44.attn_q_b.weight with CPU loading blk.44.attn_kv_a_mqa.weight with CPU loading blk.44.attn_kv_a_norm.weight to cuda:0 loading blk.44.attn_kv_b.weight to cuda:0 loading blk.44.attn_output.weight with CPU loading blk.44.ffn_gate_shexp.weight with CPU loading blk.44.ffn_up_shexp.weight with CPU loading blk.44.ffn_down_shexp.weight with CPU loading blk.44.attn_norm.weight to cuda:0 loading blk.44.ffn_norm.weight to cuda:0 loading blk.45.attn_q_a.weight with CPU loading blk.45.attn_q_a_norm.weight to cuda:0 loading blk.45.attn_q_b.weight with CPU loading blk.45.attn_kv_a_mqa.weight with CPU loading blk.45.attn_kv_a_norm.weight to cuda:0 loading blk.45.attn_kv_b.weight to cuda:0 loading blk.45.attn_output.weight with CPU loading blk.45.ffn_gate_shexp.weight with CPU loading blk.45.ffn_up_shexp.weight with CPU loading blk.45.ffn_down_shexp.weight with CPU loading blk.45.attn_norm.weight to cuda:0 loading blk.45.ffn_norm.weight to cuda:0 loading blk.46.attn_q_a.weight with CPU loading blk.46.attn_q_a_norm.weight to cuda:0 loading blk.46.attn_q_b.weight with CPU loading blk.46.attn_kv_a_mqa.weight with CPU loading blk.46.attn_kv_a_norm.weight to cuda:0 loading blk.46.attn_kv_b.weight to cuda:0 loading blk.46.attn_output.weight with CPU loading blk.46.ffn_gate_shexp.weight with CPU loading blk.46.ffn_up_shexp.weight with CPU loading blk.46.ffn_down_shexp.weight with CPU loading blk.46.attn_norm.weight to cuda:0 loading blk.46.ffn_norm.weight to cuda:0 loading blk.47.attn_q_a.weight with CPU loading blk.47.attn_q_a_norm.weight to cuda:0 loading blk.47.attn_q_b.weight with CPU loading blk.47.attn_kv_a_mqa.weight with CPU loading blk.47.attn_kv_a_norm.weight to cuda:0 loading blk.47.attn_kv_b.weight to cuda:0 loading blk.47.attn_output.weight with CPU loading blk.47.ffn_gate_shexp.weight with CPU loading blk.47.ffn_up_shexp.weight with CPU loading blk.47.ffn_down_shexp.weight with CPU loading blk.47.attn_norm.weight to cuda:0 loading blk.47.ffn_norm.weight to cuda:0 loading blk.48.attn_q_a.weight with CPU loading blk.48.attn_q_a_norm.weight to cuda:0 loading blk.48.attn_q_b.weight with CPU loading blk.48.attn_kv_a_mqa.weight with CPU loading blk.48.attn_kv_a_norm.weight to cuda:0 loading blk.48.attn_kv_b.weight to cuda:0 loading blk.48.attn_output.weight with CPU loading blk.48.ffn_gate_shexp.weight with CPU loading blk.48.ffn_up_shexp.weight with CPU loading blk.48.ffn_down_shexp.weight with CPU loading blk.48.attn_norm.weight to cuda:0 loading blk.48.ffn_norm.weight to cuda:0 loading blk.49.attn_q_a.weight with CPU loading blk.49.attn_q_a_norm.weight to cuda:0 loading blk.49.attn_q_b.weight with CPU loading blk.49.attn_kv_a_mqa.weight with CPU loading blk.49.attn_kv_a_norm.weight to cuda:0 loading blk.49.attn_kv_b.weight to cuda:0 loading blk.49.attn_output.weight with CPU loading blk.49.ffn_gate_shexp.weight with CPU loading blk.49.ffn_up_shexp.weight with CPU loading blk.49.ffn_down_shexp.weight with CPU loading blk.49.attn_norm.weight to cuda:0 loading blk.49.ffn_norm.weight to cuda:0 loading blk.50.attn_q_a.weight with CPU loading blk.50.attn_q_a_norm.weight to cuda:0 loading blk.50.attn_q_b.weight with CPU loading blk.50.attn_kv_a_mqa.weight with CPU loading blk.50.attn_kv_a_norm.weight to cuda:0 loading blk.50.attn_kv_b.weight to cuda:0 loading blk.50.attn_output.weight with CPU loading blk.50.ffn_gate_shexp.weight with CPU loading blk.50.ffn_up_shexp.weight with CPU loading blk.50.ffn_down_shexp.weight with CPU loading blk.50.attn_norm.weight to cuda:0 loading blk.50.ffn_norm.weight to cuda:0 loading blk.51.attn_q_a.weight with CPU loading blk.51.attn_q_a_norm.weight to cuda:0 loading blk.51.attn_q_b.weight with CPU loading blk.51.attn_kv_a_mqa.weight with CPU loading blk.51.attn_kv_a_norm.weight to cuda:0 loading blk.51.attn_kv_b.weight to cuda:0 loading blk.51.attn_output.weight with CPU loading blk.51.ffn_gate_shexp.weight with CPU loading blk.51.ffn_up_shexp.weight with CPU loading blk.51.ffn_down_shexp.weight with CPU loading blk.51.attn_norm.weight to cuda:0 loading blk.51.ffn_norm.weight to cuda:0 loading blk.52.attn_q_a.weight with CPU loading blk.52.attn_q_a_norm.weight to cuda:0 loading blk.52.attn_q_b.weight with CPU loading blk.52.attn_kv_a_mqa.weight with CPU loading blk.52.attn_kv_a_norm.weight to cuda:0 loading blk.52.attn_kv_b.weight to cuda:0 loading blk.52.attn_output.weight with CPU loading blk.52.ffn_gate_shexp.weight with CPU loading blk.52.ffn_up_shexp.weight with CPU loading blk.52.ffn_down_shexp.weight with CPU loading blk.52.attn_norm.weight to cuda:0 loading blk.52.ffn_norm.weight to cuda:0 loading blk.53.attn_q_a.weight with CPU loading blk.53.attn_q_a_norm.weight to cuda:0 loading blk.53.attn_q_b.weight with CPU loading blk.53.attn_kv_a_mqa.weight with CPU loading blk.53.attn_kv_a_norm.weight to cuda:0 loading blk.53.attn_kv_b.weight to cuda:0 loading blk.53.attn_output.weight with CPU loading blk.53.ffn_gate_shexp.weight with CPU loading blk.53.ffn_up_shexp.weight with CPU loading blk.53.ffn_down_shexp.weight with CPU loading blk.53.attn_norm.weight to cuda:0 loading blk.53.ffn_norm.weight to cuda:0 loading blk.54.attn_q_a.weight with CPU loading blk.54.attn_q_a_norm.weight to cuda:0 loading blk.54.attn_q_b.weight with CPU loading blk.54.attn_kv_a_mqa.weight with CPU loading blk.54.attn_kv_a_norm.weight to cuda:0 loading blk.54.attn_kv_b.weight to cuda:0 loading blk.54.attn_output.weight with CPU loading blk.54.ffn_gate_shexp.weight with CPU loading blk.54.ffn_up_shexp.weight with CPU loading blk.54.ffn_down_shexp.weight with CPU loading blk.54.attn_norm.weight to cuda:0 loading blk.54.ffn_norm.weight to cuda:0 loading blk.55.attn_q_a.weight with CPU loading blk.55.attn_q_a_norm.weight to cuda:0 loading blk.55.attn_q_b.weight with CPU loading blk.55.attn_kv_a_mqa.weight with CPU loading blk.55.attn_kv_a_norm.weight to cuda:0 loading blk.55.attn_kv_b.weight to cuda:0 loading blk.55.attn_output.weight with CPU loading blk.55.ffn_gate_shexp.weight with CPU loading blk.55.ffn_up_shexp.weight with CPU loading blk.55.ffn_down_shexp.weight with CPU loading blk.55.attn_norm.weight to cuda:0 loading blk.55.ffn_norm.weight to cuda:0 loading blk.56.attn_q_a.weight with CPU loading blk.56.attn_q_a_norm.weight to cuda:0 loading blk.56.attn_q_b.weight with CPU loading blk.56.attn_kv_a_mqa.weight with CPU loading blk.56.attn_kv_a_norm.weight to cuda:0 loading blk.56.attn_kv_b.weight to cuda:0 loading blk.56.attn_output.weight with CPU loading blk.56.ffn_gate_shexp.weight with CPU loading blk.56.ffn_up_shexp.weight with CPU loading blk.56.ffn_down_shexp.weight with CPU loading blk.56.attn_norm.weight to cuda:0 loading blk.56.ffn_norm.weight to cuda:0 loading blk.57.attn_q_a.weight with CPU loading blk.57.attn_q_a_norm.weight to cuda:0 loading blk.57.attn_q_b.weight with CPU loading blk.57.attn_kv_a_mqa.weight with CPU loading blk.57.attn_kv_a_norm.weight to cuda:0 loading blk.57.attn_kv_b.weight to cuda:0 loading blk.57.attn_output.weight with CPU loading blk.57.ffn_gate_shexp.weight with CPU loading blk.57.ffn_up_shexp.weight with CPU loading blk.57.ffn_down_shexp.weight with CPU loading blk.57.attn_norm.weight to cuda:0 loading blk.57.ffn_norm.weight to cuda:0 loading blk.58.attn_q_a.weight with CPU loading blk.58.attn_q_a_norm.weight to cuda:0 loading blk.58.attn_q_b.weight with CPU loading blk.58.attn_kv_a_mqa.weight with CPU loading blk.58.attn_kv_a_norm.weight to cuda:0 loading blk.58.attn_kv_b.weight to cuda:0 loading blk.58.attn_output.weight with CPU loading blk.58.ffn_gate_shexp.weight with CPU loading blk.58.ffn_up_shexp.weight with CPU loading blk.58.ffn_down_shexp.weight with CPU loading blk.58.attn_norm.weight to cuda:0 loading blk.58.ffn_norm.weight to cuda:0 loading blk.59.attn_q_a.weight with CPU loading blk.59.attn_q_a_norm.weight to cuda:0 loading blk.59.attn_q_b.weight with CPU loading blk.59.attn_kv_a_mqa.weight with CPU loading blk.59.attn_kv_a_norm.weight to cuda:0 loading blk.59.attn_kv_b.weight to cuda:0 loading blk.59.attn_output.weight with CPU loading blk.59.ffn_gate_shexp.weight with CPU loading blk.59.ffn_up_shexp.weight with CPU loading blk.59.ffn_down_shexp.weight with CPU loading blk.59.attn_norm.weight to cuda:0 loading blk.59.ffn_norm.weight to cuda:0 loading blk.60.attn_q_a.weight with CPU loading blk.60.attn_q_a_norm.weight to cuda:0 loading blk.60.attn_q_b.weight with CPU loading blk.60.attn_kv_a_mqa.weight with CPU loading blk.60.attn_kv_a_norm.weight to cuda:0 loading blk.60.attn_kv_b.weight to cuda:0 loading blk.60.attn_output.weight with CPU loading blk.60.ffn_gate_shexp.weight with CPU loading blk.60.ffn_up_shexp.weight with CPU loading blk.60.ffn_down_shexp.weight with CPU loading blk.60.attn_norm.weight to cuda:0 loading blk.60.ffn_norm.weight to cuda:0 loading output_norm.weight to cuda:0 loading output to cuda using CPUInfer Chat: hello /home/ubuntu/repos/kvcache-ai/ktransformers/ktransformers/operators/RoPE.py:269: UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:310.) freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2) malloc(): unaligned tcache chunk detected Aborted (core dumped) ``` and somtimes instead of malloc error, it show ``` Segmentation fault (core dumped) ``` ### Reproduction ``` /home/ubuntu/pythons/python-3.12.9/bin/python3.12 -m ktransformers.local_chat --model_path /home/ubuntu/resources/models/deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntu/resources/models/unsloth/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf --optimize_config_path /home/ubuntu/repos/kvcache-ai/ktransformers/ktransformers/optimize/optimize_rules/rocm/DeepSeek-V3-Chat.yaml --cpu_infer 28 ``` ### Environment | - | info | | --- | --- | | OS | Ubuntu24.04 | | CPU | AMD 7F52 | | GPU | AMD 7900XTX | | RAM | 256G 2666 |
bsakaguchi commented 2026-10-29 14:42:26 +00:00
Author
Owner
Copy link

@Ryuukeisyou commented on GitHub (Mar 17, 2025):

well, I pull the code, unsintalled rocm through amdgpu-uninstall and then installed a newer version, the problem is gone.
not sure which part had been the cause but anyway it is working now.

prompt eval count:    6 token(s)
prompt eval duration: 2.5350630283355713s
prompt eval rate:     2.366805058862532 tokens/s
eval count:           73 token(s)
eval duration:        44.400869846343994s
eval rate:            1.6441119341271393 tokens/s

Thank you again for your nice works!

<!-- gh-comment-id:2729125257 --> @Ryuukeisyou commented on GitHub (Mar 17, 2025): well, I pull the code, unsintalled rocm through ```amdgpu-uninstall``` and then installed a newer version, the problem is gone. not sure which part had been the cause but anyway it is working now. ``` prompt eval count: 6 token(s) prompt eval duration: 2.5350630283355713s prompt eval rate: 2.366805058862532 tokens/s eval count: 73 token(s) eval duration: 44.400869846343994s eval rate: 1.6441119341271393 tokens/s ``` Thank you again for your nice works!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#649
No description provided.