Deepgram streaming some times giving no response

[samueljazzjohn]

Which Deepgram product are you using?

Deepgram SDKs

Details

When I use the Deepgram live streaming function to perform speech-to-text (STT) conversion, it does not provide accurate transcriptions. It seems to be missing some text. I am also adding the code here. Please help me to find the root cause of that. Also I would like to know that can we get the audio directly from microphone using deepgram real-time transcription.

If you are making a request to the Deepgram API, what is the full Deepgram URL you are making a request to?

No response

If you are making a request to the Deepgram API and have a request ID, please paste it below:

No response

If possible, please attach your code or paste it into the text box.

transcription_data = []  # To store the transcription data

def handle_transcript(transcript):
    transcription_data.append(transcript)  # Store the transcript in the list

async def recognize_speech_deepgram_live():
    r = sr.Recognizer()
    while True:
        try:
            audio_file, randum_chuck, i = audio_queue.get()

            with sr.AudioFile(audio_file) as source:
                audio_data = r.record(source)

            try:
                start_time = time.time()
                try:
                    deepgramLive = await dg_client.transcription.live({
                        'panctuation': True,
                        'smart_format': True,
                        'interim_results': False,
                        'language': 'en-US',
                        'model': 'nova',
                    })
                except Exception as e:
                    print(f'Could not open socket: {e}')
                
                deepgramLive.registerHandler(deepgramLive.event.CLOSE, lambda c: print(f'Connection closed with code {c}.'))
                deepgramLive.registerHandler(deepgramLive.event.TRANSCRIPT_RECEIVED, handle_transcript)

                deepgramLive.send(audio_data.get_wav_data())

                transcription_text = ''
                
                for transcript in transcription_data:
                    transcription_text = transcript['channel']['alternatives'][0]['transcript']

                print(transcription_text)

            except sr.UnknownValueError:
                logger.debug("Could not understand audio")
            except sr.RequestError as e:
                logger.error(e)
        except Exception as e:
            logger.exception(e)
            sleep(0.01)
        except:
            logger.error(traceback.format_exc())
            sleep(0.01)

If possible, please attach an example audio file to reproduce the issue.

No response

[shirgoldbird]

Hi @samueljazzjohn! What type of inaccurate results are you receiving from Deepgram? Could you expand on what you mean by "missing some text"?

Without knowing more about your audio source and what Deepgram is returning, it's hard to say what might be going wrong. Some things that can cause inaccurate or incomplete transcriptions: poor quality audio, accidentally truncated audio, audio with background noise, multilingual audio, audio that DG's models haven't been trained on, etc. You can start by listening to the audio you're sending Deepgram to see if there are any characteristics present that might interfere with ASR. Or if you want to send a sample, we can listen and offer our opinion. :)

Deepgram's streaming test suite (guide, GitHub) has an example of streaming audio from the microphone, as well as several other actionable steps for getting started with streaming. I'd recommend following the guide and comparing the quality of results from our audio sample + code to the results you're getting. This may shed some light on the issue.

Let us know how it goes.

[samueljazzjohn]

I have tested the same audio using the Deepgram prerecorded Python SDK, and it yielded an accuracy rate of 98%, which is impressive. I'm not sure if I'm following the correct approach in streaming SDK. By the way, could you please guide me on how to attatch the audio file?

[shirgoldbird]

@samueljazzjohn Accuracy varies between pre-recorded and live streaming transcription. Live streaming may result in less accurate transcriptions, as for pre-recorded transcriptions, Deepgram's models have the benefit of having the entire audio context to work with.

You can provide us with request IDs, which will allow us to investigate your audio further, or attach files via GitHub.

[samueljazzjohn]

try:
                    deepgramLive = await dg_client.transcription.live({
                        'panctuation': True,
                        'smart_format': True,
                        'interim_results': False,
                        'language': 'en-US',
                        'model': 'nova',
                    })
                except Exception as e:
                    print(f'Could not open socket: {e}')
                
                deepgramLive.registerHandler(deepgramLive.event.CLOSE, lambda c: print(f'Connection closed with code {c}.'))
                deepgramLive.registerHandler(deepgramLive.event.TRANSCRIPT_RECEIVED, handle_transcript)

                deepgramLive.send(audio_data.get_wav_data())

I am currently using this code snippet for live speech-to-text (STT) conversion. How can I retrieve the transcription when a speaker or sentence is completed also?

[samueljazzjohn]

if transcript['type'] == 'Results':
        transcript = transcript['channel']['alternatives'][0]['transcript']
        print("Transcription:", transcript)

Is it a right way to get the transcript?

[jjmaldonis]

Yes this is correct

[samueljazzjohn]

test_deepgram_live.csv

This is the sheet it generating. I am actually using the seperate thread for audio recording and seperate thread for transcription. the recorded chunks of audio will be stored in queue and the transcription thread will convert that audio data to text.

I am also adding the exact transcritption from that audio.

0:02
And I will start walking through my understanding my interpretation of about the requirements. Yeah, yeah, OK, and I have the original proposal appearing on my left. Is the font size OK or should I increase more?
0:30
Yeah. If you increase it a bit, please, that would be great. OK. Sure. Is that OK? Yeah, that's great. Thanks. OK. So the problem we are trying to here find out is that I use the word pro here. And what I mean by a Pro is that there is a human who knows a lot of answers related to a domain. The domain is WIG.
1:02
So we are this. This is a very similar situation we are in these days with other few clients also tie, which means that for example, give an example of another client. This guy is a pro of swimming pools, yeah, and he manufacturers equipments which are used to maintain the swimming pool, ranging from cleanliness to aesthetics and more.
1:26
So he's like a pro on what chemicals what maintenance etc and he has a lot of questions answers diagnosis going on likewise you know you you being pro or you having people in your contact who are professional in helping people make decisions related to wigs. I think yeah time yes yes. OK so customer is asking a lot of questions to a human Vic professional and.
1:56
You find a pattern in here and you especially with the advent of generative AI like ChatGPT, we all have started to find that Okay, lots of these questions repetitive or not they could be answered by a chat bot. Yeah okay. Yeah and and so the solution is we would have a chat bot it would basically.
2:24
People would be able to get answers to these their questions and the experience should be as close as talking to a real human who is who actually trained the chat bot. So which means that we need to build the software not just for the people who are coming in to seek advice related to wigs. We need to build the software for you as well so that you as a wig pro can continue to stay in the loop.

Can we transcript directly from the mic without running the reperate thread

[jjmaldonis]

Two threads are required in this situation. If only one thread was used, the mic would stop recording while the audio is being transcribed.

[samueljazzjohn]

In this scenario, I believe a significant number of words are being omitted. Could you please assist me in identifying the root cause of this issue?

[jjmaldonis]

Sure, can you upload the audio you're using? You can click on the "Attach files by dragging & dropping, ..." text at the bottom of the input box (you have to click inside it). You will need to zip your audio file before uploading it.

[samueljazzjohn]

test_audio_2_out_2.wav.zip
This is the audio file...Can you please check this?