Multilingual Transcription with Diarization

[anilshanbhag]

I am interested in transcribing audio files with users speaking in English and Spanish.
Example video: https://www.youtube.com/watch?v=tFsKulFcauY

When I tried Whisper, it did a reasonable job of giving all the text. However, it is missing diarization.

I am quite excited about the multilingual feature in Nova 2.
However, when I tried it, the model seems to struggle.

Request ID: e2795fbd-b60a-4639-a3bc-17cdb91ea1ba

Nova2 output
¿Puede decirle a las señoras y señores qué fue lo que sucedió, por favor? Estábamos sentados afuera echándonos unos tragos, fumando, usted sabe, haciendo cosas que no deberíamos estar haciendo en realidad. Estábamos sentados ahí atrás comiendo plátanos y otras bobadas. Weren't supposed to be doing. We were sitting back there having plantains and other stuff. Entra en detalle, por favor. Dígale a las señoras y señores qué es lo que estaba fumando. Estábamos fumando marihuana. Yo estaba fumando con unos amigos y acababa de salir del trabajo porque salí un poquito más temprano ya que yo soy capataz. Because I'm the foreman. Dígale, por favor, a las señoras y señores qué fue lo que sucedió a partir de ese momento. Bueno, estábamos sentados ahí reunidos, conversando, usted sabe, simplemente pasándolo como la gente común y corriente hace todos los días. Bueno, y yo estaba fumando marihuana, tomando mezcal en esos momentos. Bueno, and I was smoking marihuana and drinking mezcal at that point in time. Y alguien gritó, dijo, mierda, es la chota, y no sé de dónde salió la policía, pero de repente estaban ahí corriendo hacia nosotros. And someone screamed, Shit, it's the cops. And I don't know where the police came from, but suddenly they ran toward us. And I let the booze drop and the joint drop. I dropped the marijuana I had in my hand and ran.

Whisper output
Can you tell the ladies and gentlemen what happened, please? ¿Puede decirle a las señoras y señores qué fue lo que sucedió, por favor? Estábamos sentados afuera echándonos unos tragos, fumando. Usted sabe, haciendo cosas que no deberíamos estar haciendo en realidad. Estábamos sentados allá atrás comiendo plátanos y otras bobadas. We were sitting there. We were drinking. We were smoking. We were doing things that we weren't supposed to be doing. Sitting back there, having plantains and other stuff. Would you please be specific? Would you tell the ladies and gentlemen what you were smoking? Entra en detalle, por favor. Dígale a las señoras y señores qué es lo que estaba fumando. Estábamos fumando marihuana. Yo estaba fumando con unos amigos y acababa de salir del trabajo porque salió un poquito más temprano ya que yo soy capataz. We were smoking marijuana. I was smoking with some friends. I had gotten out of work a little early because I'm the foreman. Can you tell the ladies and gentlemen what happened from that point forward? Dígale, por favor, a las señoras y señores qué fue lo que sucedió a partir de ese momento. Bueno, estábamos sentados ahí reunidos, conversando. Usted sabe, simplemente pasándolo como la gente común y corriente hace todos los días. Well, we were there. We were sitting, gathered together, chatting the way people do normally. Bueno, y yo estaba fumando marihuana, tomando mezcal en esos momentos. Well, and I was smoking marijuana and drinking mezcal at that point in time. Y alguien gritó, dijo, mierda, es la chota. Y no sé de dónde salió la policía, pero de repente estaban ahí corriendo hacia nosotros. And someone screamed, shit, it's the cops. And I don't know where the police came from, but suddenly they ran toward us. Y dejé caer el vaso de guaro y el pitillo que yo tenía, el pitillo que yo tenía en la mano, la marihuana que yo tenía en la mano, y la tiré y corrí. And I let the booze drop and the joint drop. I dropped the marijuana I had in my hand and ran.

As you can see, Nova2 misses the very first sentence: Can you tell the ladies and gentlemen what happened, please?

Any suggestions on how to improve the accuracy / solve this use case?

[jpvajda]

@anilshanbhag unfortunately there isn't much you can do yourself to improve the model's output in terms of accuracy. This is a new feature of our Nova-2 model, so it's possible you are just experiencing some of the limitations of it.

However here are some things to try to see if you can get some incremental gains:

To improve the accuracy of your audio transcription with Deepgram, you can consider the following approaches:

1. Use Keyword Boosting: This feature allows you to specify words that Deepgram should pay special attention to during transcription. It's particularly useful for uncommon words, proper nouns, product names, and industry-specific terminology. You can send up to 200 keywords per API request Deepgram Docs.

2. Choose the Right Model: Deepgram offers various models optimized for different use cases. For general purposes, 'nova-2-general' is recommended, while 'nova-2-phonecall' is better for lower-quality or phonecall audio. There are also domain-specific models for medical, finance, and other fields GitHub Discussion.

3. Consider Custom Model Training: If you have more than 200 unique keywords or phrases that aren't accurately identified with Keyword Boosting, you might want to upgrade to a trained AI model. Deepgram can train a model using your sample audio and ground truth transcripts to learn your specific language, accents, dialects, and terminology Deepgram Docs.

4. Use Smart Formatting: This feature helps make the transcript more readable and works well on most abbreviations, alphanumerics, emails, currency, and other formatting Deepgram Help Center.

5. Improve Diarization for Short Audio: If you're transcribing short audio files with multiple speakers, consider prepending a 30-second clip of the primary speaker's voice before the full audio. This can help the model better differentiate between speakers Deepgram Help Center.

Remember to benchmark your transcription results without any enhancements first, then add improvements one by one to notice the effect. Each of these methods can contribute to improving your transcription accuracy, depending on your specific use case and audio characteristics.