[GH-ISSUE #16] q5_k_m is not supported? #12

New issue

Closed

opened 2026-10-29 14:32:38 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:32:38 +00:00

Owner

Originally created by @keyonzeng on GitHub (Jul 31, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/16

I have checked custom_gguf.py and find that q5_k_m is not supported because function dequantize_q5_k_gpu does nothing.

so what is the problem with supporting q5_k_m model?

Originally created by @keyonzeng on GitHub (Jul 31, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/16 I have checked custom_gguf.py and find that q5_k_m is not supported because function dequantize_q5_k_gpu does nothing. <img width="225" alt="image" src="https://github.com/user-attachments/assets/eb1ee5cc-ca50-4662-9f02-d29b9ad37c49"> so what is the problem with supporting q5_k_m model?

bsakaguchi closed this issue

2026-10-29 14:32:38 +00:00

bsakaguchi commented

2026-10-29 14:32:39 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Jul 31, 2024):

Yes, we have supported all cpu dequant, but q5km's gpu dequant is on the way, only Q4km and Q8_0 gpu dequant are supported now. If you really need q5k I can open cpu dequant for you, but it will be 10x slower when you loading your model.

@Azure-Tang commented on GitHub (Jul 31, 2024): Yes, we have supported all cpu dequant, but q5km's gpu dequant is on the way, only Q4km and Q8_0 gpu dequant are supported now. If you really need q5k I can open cpu dequant for you, but it will be 10x slower when you loading your model.

bsakaguchi commented

2026-10-29 14:32:39 +00:00

Author

Owner

@BITcyman commented on GitHub (Aug 16, 2024):

So sorry to let you wait so long. We have supported q5_k dequant on gpu now. Other dequant types (q2_k and q3_k) on gpu have been supported as well. You can find more details in ktransformers/utils/custom_gguf.py and ktransformers/ktransformers_ext/cuda/custom_gguf/dequant.cu.

@BITcyman commented on GitHub (Aug 16, 2024): So sorry to let you wait so long. We have supported q5_k dequant on gpu now. Other dequant types (q2_k and q3_k) on gpu have been supported as well. You can find more details in ktransformers/utils/custom_gguf.py and ktransformers/ktransformers_ext/cuda/custom_gguf/dequant.cu.

bsakaguchi commented

2026-10-29 14:32:39 +00:00

Author

Owner

@344303947 commented on GitHub (Mar 17, 2025):

Help: how much extra memory is needed for q5_k_m to run in addition to the model 443GB, and can 512GB run?
求助：q5_k_m 运行除了模型443GB 还需要多少额外的内存，512GB能run起来？

@344303947 commented on GitHub (Mar 17, 2025): Help: how much extra memory is needed for q5_k_m to run in addition to the model 443GB, and can 512GB run? 求助：q5_k_m 运行除了模型443GB 还需要多少额外的内存，512GB能run起来？

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871