Stream.cpp not hearing me

[Pigarian]

Hello. I've gotten Stream.cpp to compile and run, but once it says "Start speaking" it doesn't seem to do anything. My computer notifies me an app is using my microphone, but processing isn't happening. Below is everything it prints on launch:

init: found 1 capture devices:
init:    - Capture device #0: 'Microphone Array (Realtek(R) Audio)'
init: attempt to open default capture device ...
init: obtained spec for input device (SDL Id = 2):
init:     - sample rate:       16000
init:     - format:            33056 (required: 33056)
init:     - channels:          1 (required: 1)
init:     - samples per frame: 1024
whisper_init_from_file_no_state: loading model from 'ggml-base.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: f16           = 1
whisper_model_load: type          = 2
whisper_model_load: mem required  =  218.00 MB (+    6.00 MB per decoder)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: model ctx     =  140.60 MB
whisper_model_load: model size    =  140.54 MB
whisper_init_state: kv self size  =    5.25 MB
whisper_init_state: kv cross size =   17.58 MB

main: processing 48000 samples (step = 3.0 sec / len = 10.0 sec / keep = 0.2 sec), 4 threads, lang = en, task = transcribe, timestamps = 0 ...
main: n_new_line = 2, no_context = 1

[Start speaking]

If anyone's got a good guess as to what's wrong, I'm all ears.

[DylPorter]

Going to have to bump this one, I'm facing exactly the same issue and am not sure what the problem is.

The program clearly detects a capture device, but doesn't recognise any audio input whatsoever. I've tried different capture devices, and have verified that they are all working well in pavucontrol, etc. I have also experimented with -vth N and -fth N but no luck. The web versions work absolutely fine, it's just the local install so I'm guessing it's a library problem.

The "command" and "stream" documentation suggests that the audio detection comes from SDL2, though I have the latest version installed (2.30.3) so I'm pretty stumped. Any ideas or suggestions would be greatly appreciated!

[DylPorter]

Figured out the problem, seems pretty obvious now looking back at my post...

SDL2 version 2.30+ doesn't seem to be working with whisper.cpp, and everything seems to be working fine after downgrading back to 2.28.5

Might be worth a try if you've recently updated your SDL2 library @Pigarian !

[tamo]

The log between 2.28.5 and 2.23.0 shows many changes are related to pulseaudio.
And whisper.cpp-1.6.0 on windows works well with SDL2-2.30.3.
So information about pulseaudio (at least version number?) might help someone.

[tamo]

I've found a suspicious issue too: libsdl-org/SDL#9706

[DylPorter]

The log between 2.28.5 and 2.23.0 shows many changes are related to pulseaudio.

Thank you so much for having a look, I think this makes more sense. My apologies for leaving out the detail that I'm running a Linux system. I currently have pulseaudio 17.0-3 installed on my system, hope this helps anyone who stumbles across the same problem!

[DylPorter]

I've found a suspicious issue too: libsdl-org/SDL#9706

Yes, in the log that you provided, there are a few commits related to a threaded main-loop, which corresponds with the problem described in your linked issue.

I think we can mark it down as a notable incompatibility between these two libraries, and try to investigate further?

[ggerganov]

On whisper.cpp side, we might want to add checks in the examples for the incompatible SDL versions and print some warning/errors