3 Commits

Author	SHA1	Message	Date
Iwan Kawrakow	e73ae1f6d3	bitnet(scale in a separate tensor): mul -> scale on CUDA ... On CUDA we do not have access to the tensor data until we hit the kernel. That's why this hack. In any case, iq2_bn goes back up to 228 t/s, which is close to the 234 t/s we have without the extra scale operation. PP is 9400 t/s, down from 9600 t/s, but better than the 9200 t/s we get without making the mul -> scale replacement.	2024-06-22 12:02:52 +03:00
slaren	0d56246f4b	ggml : group all experts in a single ggml_mul_mat_id (#6505) ... * ggml : group all experts in a single ggml_mul_mat_id cuda : improve mmid row copy * cuda : fix bin bcast with non-cont src0 * test-backend-ops : only run all mul mat tests for base types * llama : disable moe offloading with SYCL --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-04-18 15:18:48 +02:00
slaren	ae1f211ce2	cuda : refactor into multiple files (#6269)	2024-03-25 13:50:23 +01:00