[GH-ISSUE #87] are marline and q4k totally equivalent? #44

New issue

Closed

opened 2026-10-29 14:32:58 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:32:58 +00:00

Owner

Originally created by @Eutenacity on GitHub (Sep 13, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/87

sorry i am not familiar with quantization. i found you transfer q4k into fp32 and then quantize to 4 bit with marline. so i am curious whether marline and q4k are totally equivalent? dose this kind of transfer affect accuracy

Originally created by @Eutenacity on GitHub (Sep 13, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/87 sorry i am not familiar with quantization. i found you transfer q4k into fp32 and then quantize to 4 bit with marline. so i am curious whether marline and q4k are totally equivalent? dose this kind of transfer affect accuracy

bsakaguchi closed this issue

2026-10-29 14:32:58 +00:00

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Sep 14, 2024):

This approach will inevitably result in some loss of precision, but I believe it won’t have a significant impact, especially since we’re already using Q4 quantization.

@Azure-Tang commented on GitHub (Sep 14, 2024): This approach will inevitably result in some loss of precision, but I believe it won’t have a significant impact, especially since we’re already using Q4 quantization.

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@Eutenacity commented on GitHub (Sep 19, 2024):

ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on.

@Eutenacity commented on GitHub (Sep 19, 2024): ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on.

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Sep 23, 2024):

ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on.

The challenge is that GGML’s matrix multiplication is deeply intertwined with its memory allocation system, making it difficult to extract and reuse separately.

@Azure-Tang commented on GitHub (Sep 23, 2024): > ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on. The challenge is that GGML’s matrix multiplication is deeply intertwined with its memory allocation system, making it difficult to extract and reuse separately.

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@Eutenacity commented on GitHub (Oct 7, 2024):

ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on.

The challenge is that GGML’s matrix multiplication is deeply intertwined with its memory allocation system, making it difficult to extract and reuse separately.

I try to use ggml matmul in the pytorch. I put my code in the https://github.com/Eutenacity/python-ggml-matmul
I hope this can help.
But I can not use PyTorch cudaprah to capture the matmul.

@Eutenacity commented on GitHub (Oct 7, 2024): > > ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on. > > The challenge is that GGML’s matrix multiplication is deeply intertwined with its memory allocation system, making it difficult to extract and reuse separately. I try to use ggml matmul in the pytorch. I put my code in the https://github.com/Eutenacity/python-ggml-matmul I hope this can help. But I can not use PyTorch cudaprah to capture the matmul.

bsakaguchi commented

2026-10-29 14:33:00 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Oct 8, 2024):

ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on.

The challenge is that GGML’s matrix multiplication is deeply intertwined with its memory allocation system, making it difficult to extract and reuse separately.

I try to use ggml matmul in the pytorch. I put my code in the https://github.com/Eutenacity/python-ggml-matmul I hope this can help. But I can not use PyTorch cudaprah to capture the matmul.

Not being able to use cudagraph significantly slows down the generation speed. Let’s see if we can figure this out.

@Azure-Tang commented on GitHub (Oct 8, 2024): > > > ggml also provide fast matmul cuda kernel. Why not use it which is exactly fit with q4_k, q4_m and so on. > > > > > > The challenge is that GGML’s matrix multiplication is deeply intertwined with its memory allocation system, making it difficult to extract and reuse separately. > > I try to use ggml matmul in the pytorch. I put my code in the https://github.com/Eutenacity/python-ggml-matmul I hope this can help. But I can not use PyTorch cudaprah to capture the matmul. Not being able to use cudagraph significantly slows down the generation speed. Let’s see if we can figure this out.

bsakaguchi referenced this issue

2026-10-29 14:35:39 +00:00

[GH-ISSUE #304] 反复出现GLIBCXX_3.4.30 not found 和 CUDA error: no kernel image is available for execution on the device #193