1.2 KiB
🔀 #581 - Vulkan: Disable multi-add for now
| Author | ikawrakow |
|---|---|
| State | ❌ Closed |
| Created | 2025-07-03 |
| Updated | 2025-07-03 |
Description
...until we implement it for Vulkan, else it will run on the CPU and performance of MoE models will be terrible.
Also the Vulkan back-end has the very strange restriction that the number of experts times the number of tokens must be <= 4096 for indirect matrix multiplications (as needed in MoE models). Haven't looked into why this restriction is imposed (as I'm not familiar with the Vulkan back-end at all), so for now just using a very recent PR in mainline to split the indirect matrix multiplication into chunks, where each chunks satisfies the restriction.
But this basically means a horrible performance for MoE models. Case in point, with DeepSeek-V2-Lite I'm getting in the range of 1600 t/s PP speed (here and in mainline) vs ~9000 with the ik_llama.cpp CUDA back-end on an RTX-4080.
Curious if someone is using the Vullkan back-end in llama.cpp to run DeepSeek-V3/R1 and/or Qwen3-235B-A22B and/or LlaMA-4, etc.