[GH-ISSUE #20] If I want to run a linear layer with q4_k_m on cpu using lamafile, how to do it with your implement #11

New issue

Closed

opened 2026-10-29 14:32:38 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:32:38 +00:00

Owner

Originally created by @Eutenacity on GitHub (Aug 3, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/20

Originally assigned to: @chenht2022 on GitHub.

I found the bench_linear.py. But the weight is fp32 not the quantized uint8. Can you give me a example ? thanks

Originally created by @Eutenacity on GitHub (Aug 3, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/20 Originally assigned to: @chenht2022 on GitHub. I found the bench_linear.py. But the weight is fp32 not the quantized uint8. Can you give me a example ? thanks

bsakaguchi closed this issue

2026-10-29 14:32:38 +00:00

bsakaguchi commented

2026-10-29 14:32:39 +00:00

Author

Owner

@chenht2022 commented on GitHub (Aug 4, 2024):

You are correct. For testing convenience, we randomly generated some FP32 weights in our bench program and let llamafile_sgemm treat them as quantized weights. The output obtained this way is meaningless; we use this bench program solely to test performance. In actual use, the weights are mmap-ed from gguf rather than randomly generated, ensuring that the computation results are meaningful.

The injection of the linear layer is coming soon, and you will then be able to run a linear layer on the CPU using llamafile by modifying a YAML configuration file.

@chenht2022 commented on GitHub (Aug 4, 2024): You are correct. For testing convenience, we randomly generated some FP32 weights in our bench program and let llamafile_sgemm treat them as quantized weights. The output obtained this way is meaningless; we use this bench program solely to test performance. In actual use, the weights are mmap-ed from gguf rather than randomly generated, ensuring that the computation results are meaningful. The injection of the linear layer is coming soon, and you will then be able to run a linear layer on the CPU using llamafile by modifying a YAML configuration file.

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871