[GH-ISSUE #105] [BUG] Incorrect Version of KoboldCpp Installed via SillyTavern Toolbox for NVIDIA (CUDA) – Causes Flash Attention Error #52

New issue

Closed

opened 2026-10-15 18:06:36 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-15 18:06:36 +00:00

Owner

Originally created by @trungly1 on GitHub (May 26, 2025).
Original GitHub issue: https://github.com/SillyTavern/SillyTavern-Launcher/issues/105

Environment

🪟 Windows

System

Windows 11 Pro for Workstation - Version 24H2 (Build 26.100.4061)

Describe the problem

Issue Summary:
When installing KoboldCpp via the SillyTavern Toolbox under the NVIDIA GPU option, the application incorrectly downloads the non-CUDA version of koboldcpp.exe instead of the CUDA-enabled koboldcpp_cu12.exe. This results in Flash Attention not working properly and produces a CUDA compatibility error at runtime.

Steps to Reproduce:

Open SillyTavern
Navigate to Toolbox > App Installer > Text Completion
Select KoboldCpp, then choose NVIDIA GPU
The installer downloads koboldcpp.exe (408MB), which is not the CUDA build
Launch any GGUF model with Flash Attention enabled
Observe the error

Expected Behavior:
The installer should download and install koboldcpp_cu12.exe (approximately 555MB), which is the correct CUDA-enabled binary for NVIDIA GPUs.

when checking the code:

REM Check if koboldcpp file exists [koboldcpp NVIDIA]
if exist "%koboldcpp_install_path%\koboldcpp.exe" (
    REM Remove koboldcpp
    echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% Removing existing koboldcpp.exe
    del "%koboldcpp_install_path%\koboldcpp.exe"
    curl -L -o "%koboldcpp_install_path%\koboldcpp.exe" "https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp.exe"
    echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% %green_fg_strong%koboldcpp updated successfully.%reset%
    pause
    goto :update_manager_text_completion
)
REM Check if koboldcpp file exists [koboldcpp AMD]
if exist "%koboldcpp_install_path%\koboldcpp_rocm.exe" (
    REM Remove koboldcpp_rocm
    echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% Removing existing koboldcpp_rocm.exe
    del "%koboldcpp_install_path%\koboldcpp_rocm.exe"
    curl -L -o "%koboldcpp_install_path%\koboldcpp_rocm.exe" "https://github.com/YellowRoseCx/koboldcpp-rocm/releases/latest/download/koboldcpp_rocm.exe"
    echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% %green_fg_strong%koboldcpp_rocm updated successfully.%reset%
    pause
    goto :update_manager_text_completion
)

It looks like it links to the regular package, not the cuda one (look at rocm).

Actual Behavior:
The non-CUDA koboldcpp.exe is installed, resulting in a runtime error when Flash Attention is used.

Error Message:

c:\a\koboldcpp\koboldcpp\ggml\src\ggml-cuda\template-instances\../fattn-mma-f16.cuh:1246: ERROR: CUDA kernel flash_attn_ext_f16 has no device code compatible with CUDA arch 750. ggml-cuda.cu was compiled for: __CUDA_ARCH_LIST__

Workaround:

Manually download the correct CUDA build (koboldcpp_cu12.exe) from the [official GitHub releases](https://github.com/LostRuins/koboldcpp/releases) page and launch it in standalone mode.

Environment:

STL: 24.1.5.0
OS: Windows 11

Additional info

Please tick the boxes

I have explained the issue clearly, and I included all relevant info
I have checked that this issue hasn't already been raised
I have checked the docs

Originally created by @trungly1 on GitHub (May 26, 2025). Original GitHub issue: https://github.com/SillyTavern/SillyTavern-Launcher/issues/105 ### Environment 🪟 Windows ### System Windows 11 Pro for Workstation - Version 24H2 (Build 26.100.4061) ### Describe the problem **Issue Summary:** When installing KoboldCpp via the SillyTavern Toolbox under the *NVIDIA GPU* option, the application incorrectly downloads the non-CUDA version of `koboldcpp.exe` instead of the CUDA-enabled `koboldcpp_cu12.exe`. This results in Flash Attention not working properly and produces a CUDA compatibility error at runtime. --- **Steps to Reproduce:** 1. Open **SillyTavern** 2. Navigate to **Toolbox** > **App Installer** > **Text Completion** 3. Select **KoboldCpp**, then choose **NVIDIA GPU** 4. The installer downloads `koboldcpp.exe` (408MB), which is **not** the CUDA build 5. Launch any GGUF model with Flash Attention enabled 6. Observe the error --- **Expected Behavior:** The installer should download and install `koboldcpp_cu12.exe` (approximately 555MB), which is the correct CUDA-enabled binary for NVIDIA GPUs. when checking the code: ``` REM Check if koboldcpp file exists [koboldcpp NVIDIA] if exist "%koboldcpp_install_path%\koboldcpp.exe" ( REM Remove koboldcpp echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% Removing existing koboldcpp.exe del "%koboldcpp_install_path%\koboldcpp.exe" curl -L -o "%koboldcpp_install_path%\koboldcpp.exe" "https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp.exe" echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% %green_fg_strong%koboldcpp updated successfully.%reset% pause goto :update_manager_text_completion ) REM Check if koboldcpp file exists [koboldcpp AMD] if exist "%koboldcpp_install_path%\koboldcpp_rocm.exe" ( REM Remove koboldcpp_rocm echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% Removing existing koboldcpp_rocm.exe del "%koboldcpp_install_path%\koboldcpp_rocm.exe" curl -L -o "%koboldcpp_install_path%\koboldcpp_rocm.exe" "https://github.com/YellowRoseCx/koboldcpp-rocm/releases/latest/download/koboldcpp_rocm.exe" echo %blue_bg%[%time%]%reset% %blue_fg_strong%[INFO]%reset% %green_fg_strong%koboldcpp_rocm updated successfully.%reset% pause goto :update_manager_text_completion ) ``` It looks like it links to the regular package, not the cuda one (look at rocm). --- **Actual Behavior:** The non-CUDA `koboldcpp.exe` is installed, resulting in a runtime error when Flash Attention is used. **Error Message:** ``` c:\a\koboldcpp\koboldcpp\ggml\src\ggml-cuda\template-instances\../fattn-mma-f16.cuh:1246: ERROR: CUDA kernel flash_attn_ext_f16 has no device code compatible with CUDA arch 750. ggml-cuda.cu was compiled for: __CUDA_ARCH_LIST__ ``` --- **Workaround:** Manually download the correct CUDA build (`koboldcpp_cu12.exe`) from the [[official GitHub releases](https://github.com/LostRuins/koboldcpp/releases)](https://github.com/LostRuins/koboldcpp/releases) page and launch it in standalone mode. --- **Environment:** STL: 24.1.5.0 OS: Windows 11 --- ### Additional info ![Image](https://github.com/user-attachments/assets/025e7670-f370-4b14-81ee-7619ac56650a) ![Image](https://github.com/user-attachments/assets/ed52377a-9c4a-4797-946d-6620d6feaa1b) ### Please tick the boxes - [x] I have explained the issue clearly, and I included all relevant info - [x] I have checked that this [issue hasn't already been raised](https://github.com/SillyTavern/SillyTavern/issues?q=is%3Aissue) - [x] I have checked the [docs](https://docs.sillytavern.app/) ![important](https://img.shields.io/badge/Important!-F6094E)

bsakaguchi

2026-10-15 18:06:36 +00:00