[GH-ISSUE #20] [BUG] SillyTavern Speech Recognition Crashes with Local Xenova/whisper-medium (and larger) Models #8

New issue

Open

opened 2026-10-15 13:27:18 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-15 13:27:18 +00:00

Owner

Originally created by @myxvcx on GitHub (Jun 20, 2025).
Original GitHub issue: https://github.com/SillyTavern/Extension-Speech-Recognition/issues/20

As i really have no idea i asked AI to write this for me, but to note is that the Speech Recognition crashes with it then again working on smaller models. i would never have the nerves for that formatting. please help me still.

GitHub Issue Draft for SillyTavern

Title: SillyTavern Crashes with Local Xenova/whisper-medium (and larger) Models Due to Audio Handling in Node.js

Body:

Describe the bug
When using the local Speech Recognition feature in SillyTavern (version [Your SillyTavern Version, e.g., 1.13.0]) with transformers.js and selecting a larger Whisper model like Xenova/whisper-medium.en (or Xenova/whisper-medium), the SillyTavern server process crashes (the console window running the server disappears or closes abruptly). This makes it impossible to use larger local Whisper models.

The Xenova/whisper-small model appears to work, or at least doesn't cause a fatal crash, though it may be encountering the same underlying issue less severely.

To Reproduce
Steps to reproduce the behavior:

Set up local Speech Recognition in SillyTavern to use transformers.js.
Attempt to use or switch the model to Xenova/whisper-medium.en (or Xenova/whisper-medium).
Initiate speech input.
Observe that the SillyTavern server process crashes.

Expected behavior
SillyTavern should be able to use Xenova/whisper-medium.en (and other sizes) for local speech recognition without crashing, similar to how Xenova/whisper-small functions.

Screenshots / Logs

Initially, before implementing a standalone test, the only symptom was the server console crashing, making it difficult to get specific error logs from SillyTavern itself for the medium model.
My earlier log for whisper-medium (dada.txt) only showed ONNX Runtime warnings which are normal.
My log for whisper-small working in SillyTavern (dsa.txt) also showed normal ONNX Runtime warnings.

Standalone Test and Root Cause Identification
To investigate, I ran Xenova/whisper-medium.en in a standalone Node.js script using @xenova/transformers.

Initial Failure: When passing an audio file path directly to the pipeline() function, the script failed with:

Error: Unable to load audio from path/URL since `AudioContext` is not available in your environment. Instead, audio data should be passed directly to the pipeline/processor. For more information and some example code, see https://huggingface.co/docs/transformers.js/guides/node-audio-processing.

Successful Tests with Pre-Processed Audio:
By modifying the script to manually load the audio file, decode it to a 16kHz mono Float32Array, and pass this buffer directly to the pipeline(), Xenova/whisper-medium.en works perfectly on my system:
- On CPU: Successful transcription.
- On GPU (via WebGPU): Successful transcription (approx. 4.09s for a 10s audio clip).
- On GPU (via CUDA): Successful transcription (approx. 3.25s for the same 10s audio clip).

This strongly suggests that SillyTavern's Speech Recognition extension is currently passing audio file paths directly to transformers.js in its Node.js environment. This is incompatible with how transformers.js handles audio file paths outside of a browser (which relies on AudioContext). The resulting error is likely causing the Node.js process to crash, especially with the higher resource demands of larger models like whisper-medium.

Working Standalone Node.js Script:
Here is the script that successfully runs Xenova/whisper-medium.en on both CPU and GPU by pre-processing the audio:

// test-whisper.mjs
import { pipeline, env } from '@xenova/transformers';
import fs from 'fs';
import wav from 'wav'; // Requires `npm install wav`

// --- Configuration for Testing (choose one env.backends.onnx.executionProviders line) ---
// For CPU:
// env.backends.onnx.executionProviders = ['cpu'];

// For WebGPU (often default GPU attempt):
// env.backends.onnx.executionProviders = ['webgpu', 'cpu'];

// For CUDA:
env.backends.onnx.executionProviders = ['cuda', 'cpu'];


async function loadAudio(filePath) {
    return new Promise((resolve, reject) => {
        const file = fs.createReadStream(filePath);
        const reader = new wav.Reader();
        let audioData = [];
        let format = null;

        reader.on('format', fmt => {
            format = fmt;
            if (fmt.sampleRate !== 16000) {
                console.warn(`Audio sample rate is ${fmt.sampleRate} Hz, Whisper expects 16000 Hz. Results may be suboptimal.`);
            }
            if (fmt.channels !== 1) {
                console.warn(`Audio has ${fmt.channels} channels, Whisper expects 1 (mono). Using first channel data if interleaved. Results may be suboptimal.`);
            }
        });

        reader.on('data', chunk => {
            if (!format) return;
            for (let i = 0; i < chunk.length; i += format.blockAlign) {
                if (format.audioFormat === 1 && format.bitDepth === 16) { // PCM 16-bit
                    const val = chunk.readInt16LE(i);
                    audioData.push(val / 32768.0);   // Normalize to -1.0 to 1.0
                }
            }
        });

        reader.on('end', () => {
            if (!format) {
                reject(new Error("Could not read WAV format."));
                return;
            }
            resolve(new Float32Array(audioData));
        });

        reader.on('error', err => {
            reject(err);
        });

        file.pipe(reader);
    });
}

async function main() {
    try {
        console.log(`Attempting to initialize pipeline for Xenova/whisper-medium.en with EPs: ${env.backends.onnx.executionProviders.join(', ')}...`);

        const transcriber = await pipeline(
            'automatic-speech-recognition',
            'Xenova/whisper-medium.en'
        );

        console.log('Pipeline initialized.');
        if (transcriber.model && transcriber.model.session && transcriber.model.session.providers) {
             console.log('ONNX Session Configured Execution Providers for this session (approx):', transcriber.model.session.providers);
        }

        const audioFilePath = '/path/to/your/16khz_mono_test.wav'; // User needs to replace this
        console.log(`Loading and processing audio from ${audioFilePath}...`);
        const audioBuffer = await loadAudio(audioFilePath);
        console.log(`Audio loaded, buffer length: ${audioBuffer.length}`);

        console.log(`Transcribing audio buffer...`);
        const startTime = performance.now();
        const output = await transcriber(audioBuffer);
        const endTime = performance.now();
        console.log('Transcription output:', output);
        console.log(`Transcription time: ${(endTime - startTime) / 1000} seconds`);

    } catch (error) {
        console.error('Error during Whisper model test:');
        console.error(error);
    }
}

main();

Suggested Fix:
The Speech Recognition extension (or the relevant part of SillyTavern handling local transformers.js ASR) needs to be updated to handle audio input correctly for Node.js environments. This involves:

Reading the audio file.
Decoding it and resampling to a 16kHz mono Float32Array.
Passing this raw audio buffer to the transformers.js pipeline, instead of a file path.

The official transformers.js documentation provides guidance on this:
https://huggingface.co/docs/transformers.js/guides/node-audio-processing

Desktop (please complete the following information):

OS: EndeavourOS (Arch Linux)
Node.js version: v24.2.0
SillyTavern version: 1.13.0 'release' (f12c523fc)
Browser (if applicable, though server crashes): Firefox 139.0
GPU: NVIDIA GeForce RTX 4070 Ti
RAM: 30.6 GiB
Relevant SillyTavern Extensions: Speech Recognition (using transformers.js locally) Extension Version 1.1.0 (main-83c38e5)

Thank you for looking into this! Proper audio handling should allow for stable use of larger local Whisper models.

here a bit of context for finding it out
https://drive.google.com/file/d/15zgVrat9k5rMfz8r6XMGMEZ1e22QWZ57/view?usp=sharing, https://drive.google.com/file/d/1LatL1YGptgwj4D7O3NZRGYOMS4fyX3iv/view?usp=sharing, https://drive.google.com/file/d/1RC48CTlqP-RXf-LnZTzw50PP_h6xRMEQ/view?usp=sharing, https://drive.google.com/file/d/1l6B6eqVI7ummen8poPTxeo-lVpaJ1wH1/view?usp=sharing, https://drive.google.com/file/d/1msWPBRwsfEWoeExEkGfAHyX2faZu2G9v/view?usp=sharing, https://drive.google.com/file/d/1pBXZdnz1eHTDpu_BzGNtAgtLL-flNMl6/view?usp=sharing, https://drive.google.com/file/d/1pMzm7xHicteHePCo001bLZHBDDoT7Y-1/view?usp=sharing, https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221vxAuZZw9kOVHRHekB2EBR_kt4lTfVA-A%22%5D,%22action%22:%22open%22,%22userId%22:%22103431721723232481461%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing

Originally created by @myxvcx on GitHub (Jun 20, 2025). Original GitHub issue: https://github.com/SillyTavern/Extension-Speech-Recognition/issues/20 As i really have no idea i asked AI to write this for me, but to note is that the Speech Recognition crashes with it then again working on smaller models. i would never have the nerves for that formatting. please help me still. --- **GitHub Issue Draft for SillyTavern** **Title:** SillyTavern Crashes with Local `Xenova/whisper-medium` (and larger) Models Due to Audio Handling in Node.js **Body:** **Describe the bug** When using the local Speech Recognition feature in SillyTavern (version `[Your SillyTavern Version, e.g., 1.13.0]`) with `transformers.js` and selecting a larger Whisper model like `Xenova/whisper-medium.en` (or `Xenova/whisper-medium`), the SillyTavern server process crashes (the console window running the server disappears or closes abruptly). This makes it impossible to use larger local Whisper models. The `Xenova/whisper-small` model appears to work, or at least doesn't cause a fatal crash, though it may be encountering the same underlying issue less severely. **To Reproduce** Steps to reproduce the behavior: 1. Set up local Speech Recognition in SillyTavern to use `transformers.js`. 2. Attempt to use or switch the model to `Xenova/whisper-medium.en` (or `Xenova/whisper-medium`). 3. Initiate speech input. 4. Observe that the SillyTavern server process crashes. **Expected behavior** SillyTavern should be able to use `Xenova/whisper-medium.en` (and other sizes) for local speech recognition without crashing, similar to how `Xenova/whisper-small` functions. **Screenshots / Logs** * Initially, before implementing a standalone test, the only symptom was the server console crashing, making it difficult to get specific error logs from SillyTavern itself for the `medium` model. * My earlier log for `whisper-medium` (`dada.txt`) only showed ONNX Runtime warnings which are normal. * My log for `whisper-small` working in SillyTavern (`dsa.txt`) also showed normal ONNX Runtime warnings. **Standalone Test and Root Cause Identification** To investigate, I ran `Xenova/whisper-medium.en` in a standalone Node.js script using `@xenova/transformers`. * **Initial Failure:** When passing an audio file path directly to the `pipeline()` function, the script failed with: ``` Error: Unable to load audio from path/URL since `AudioContext` is not available in your environment. Instead, audio data should be passed directly to the pipeline/processor. For more information and some example code, see https://huggingface.co/docs/transformers.js/guides/node-audio-processing. ``` * **Successful Tests with Pre-Processed Audio:** By modifying the script to manually load the audio file, decode it to a 16kHz mono `Float32Array`, and pass this buffer directly to the `pipeline()`, `Xenova/whisper-medium.en` works perfectly on my system: * **On CPU:** Successful transcription. * **On GPU (via WebGPU):** Successful transcription (approx. 4.09s for a 10s audio clip). * **On GPU (via CUDA):** Successful transcription (approx. 3.25s for the same 10s audio clip). This strongly suggests that SillyTavern's Speech Recognition extension is currently passing audio file paths directly to `transformers.js` in its Node.js environment. This is incompatible with how `transformers.js` handles audio file paths outside of a browser (which relies on `AudioContext`). The resulting error is likely causing the Node.js process to crash, especially with the higher resource demands of larger models like `whisper-medium`. **Working Standalone Node.js Script:** Here is the script that successfully runs `Xenova/whisper-medium.en` on both CPU and GPU by pre-processing the audio: ```javascript // test-whisper.mjs import { pipeline, env } from '@xenova/transformers'; import fs from 'fs'; import wav from 'wav'; // Requires `npm install wav` // --- Configuration for Testing (choose one env.backends.onnx.executionProviders line) --- // For CPU: // env.backends.onnx.executionProviders = ['cpu']; // For WebGPU (often default GPU attempt): // env.backends.onnx.executionProviders = ['webgpu', 'cpu']; // For CUDA: env.backends.onnx.executionProviders = ['cuda', 'cpu']; async function loadAudio(filePath) { return new Promise((resolve, reject) => { const file = fs.createReadStream(filePath); const reader = new wav.Reader(); let audioData = []; let format = null; reader.on('format', fmt => { format = fmt; if (fmt.sampleRate !== 16000) { console.warn(`Audio sample rate is ${fmt.sampleRate} Hz, Whisper expects 16000 Hz. Results may be suboptimal.`); } if (fmt.channels !== 1) { console.warn(`Audio has ${fmt.channels} channels, Whisper expects 1 (mono). Using first channel data if interleaved. Results may be suboptimal.`); } }); reader.on('data', chunk => { if (!format) return; for (let i = 0; i < chunk.length; i += format.blockAlign) { if (format.audioFormat === 1 && format.bitDepth === 16) { // PCM 16-bit const val = chunk.readInt16LE(i); audioData.push(val / 32768.0); // Normalize to -1.0 to 1.0 } } }); reader.on('end', () => { if (!format) { reject(new Error("Could not read WAV format.")); return; } resolve(new Float32Array(audioData)); }); reader.on('error', err => { reject(err); }); file.pipe(reader); }); } async function main() { try { console.log(`Attempting to initialize pipeline for Xenova/whisper-medium.en with EPs: ${env.backends.onnx.executionProviders.join(', ')}...`); const transcriber = await pipeline( 'automatic-speech-recognition', 'Xenova/whisper-medium.en' ); console.log('Pipeline initialized.'); if (transcriber.model && transcriber.model.session && transcriber.model.session.providers) { console.log('ONNX Session Configured Execution Providers for this session (approx):', transcriber.model.session.providers); } const audioFilePath = '/path/to/your/16khz_mono_test.wav'; // User needs to replace this console.log(`Loading and processing audio from ${audioFilePath}...`); const audioBuffer = await loadAudio(audioFilePath); console.log(`Audio loaded, buffer length: ${audioBuffer.length}`); console.log(`Transcribing audio buffer...`); const startTime = performance.now(); const output = await transcriber(audioBuffer); const endTime = performance.now(); console.log('Transcription output:', output); console.log(`Transcription time: ${(endTime - startTime) / 1000} seconds`); } catch (error) { console.error('Error during Whisper model test:'); console.error(error); } } main(); ``` **Suggested Fix:** The Speech Recognition extension (or the relevant part of SillyTavern handling local `transformers.js` ASR) needs to be updated to handle audio input correctly for Node.js environments. This involves: 1. Reading the audio file. 2. Decoding it and resampling to a 16kHz mono `Float32Array`. 3. Passing this raw audio buffer to the `transformers.js` pipeline, instead of a file path. The official `transformers.js` documentation provides guidance on this: [https://huggingface.co/docs/transformers.js/guides/node-audio-processing](https://huggingface.co/docs/transformers.js/guides/node-audio-processing) **Desktop (please complete the following information):** * **OS:** EndeavourOS (Arch Linux) * **Node.js version:** v24.2.0 * **SillyTavern version:** `1.13.0 'release' (f12c523fc)` * **Browser (if applicable, though server crashes):** Firefox 139.0 * **GPU:** NVIDIA GeForce RTX 4070 Ti * **RAM:** 30.6 GiB * **Relevant SillyTavern Extensions:** Speech Recognition (using `transformers.js` locally) `Extension Version 1.1.0 (main-83c38e5)` Thank you for looking into this! Proper audio handling should allow for stable use of larger local Whisper models. --- here a bit of context for finding it out https://drive.google.com/file/d/15zgVrat9k5rMfz8r6XMGMEZ1e22QWZ57/view?usp=sharing, https://drive.google.com/file/d/1LatL1YGptgwj4D7O3NZRGYOMS4fyX3iv/view?usp=sharing, https://drive.google.com/file/d/1RC48CTlqP-RXf-LnZTzw50PP_h6xRMEQ/view?usp=sharing, https://drive.google.com/file/d/1l6B6eqVI7ummen8poPTxeo-lVpaJ1wH1/view?usp=sharing, https://drive.google.com/file/d/1msWPBRwsfEWoeExEkGfAHyX2faZu2G9v/view?usp=sharing, https://drive.google.com/file/d/1pBXZdnz1eHTDpu_BzGNtAgtLL-flNMl6/view?usp=sharing, https://drive.google.com/file/d/1pMzm7xHicteHePCo001bLZHBDDoT7Y-1/view?usp=sharing, https://aistudio.google.com/app/prompts?state=%7B%22ids%22:%5B%221vxAuZZw9kOVHRHekB2EBR_kt4lTfVA-A%22%5D,%22action%22:%22open%22,%22userId%22:%22103431721723232481461%22,%22resourceKeys%22:%7B%7D%7D&usp=sharing

bsakaguchi commented

2026-10-15 13:27:19 +00:00

Author

Owner

@myxvcx commented on GitHub (Jun 22, 2025):

Scenario A (Most Likely): Malformed audio input being prepared by sillytavern-transformers due to the underlying AudioContext failure. This malformed input (e.g., wrong shape, wrong data type, NaNs, Infs, or even just zeros if loading failed completely) is then passed to OrtRun(), causing an internal ONNX operator to fail during the encoderForward pass, resulting in error code = 6 (ORT_FAIL). The tensor dump could be the problematic input tensor.

Scenario B (Less Likely, but possible if onnxruntime-web is used suboptimally): If onnxruntime-web is being used in Node.js, and the whisper-medium.en model contains an operator or a configuration that is poorly supported or buggy in the WASM backend (especially when stressed by a larger model), it could fail with ORT_FAIL. The audio loading issue might still be present but the final crash point is an incompatibility with the runtime itself for that model.

@myxvcx commented on GitHub (Jun 22, 2025): Scenario A (Most Likely): Malformed audio input being prepared by sillytavern-transformers due to the underlying AudioContext failure. This malformed input (e.g., wrong shape, wrong data type, NaNs, Infs, or even just zeros if loading failed completely) is then passed to OrtRun(), causing an internal ONNX operator to fail during the encoderForward pass, resulting in error code = 6 (ORT_FAIL). The tensor dump could be the problematic input tensor. Scenario B (Less Likely, but possible if onnxruntime-web is used suboptimally): If onnxruntime-web is being used in Node.js, and the whisper-medium.en model contains an operator or a configuration that is poorly supported or buggy in the WASM backend (especially when stressed by a larger model), it could fail with ORT_FAIL. The audio loading issue might still be present but the final crash point is an incompatibility with the runtime itself for that model.

bsakaguchi referenced this issue

2026-10-15 13:27:26 +00:00

[PR #8] [MERGED] Turn off the mic when switching providers #17