Languages getting mixed up (curiosities of multilingual models)

[Tronic]

This model has some curious properties. Being a native Finnish speaker, it picks up other languages that I speak as Finnish in language detection, presumably because of my thick Finnish accent, but actually understands the words spoken in the target language. I presume that training the model made it pick up not the language and the words spoken, but the accent of the speaker as the primary deciding factor.

Another type of mix-up is seen with forced language, e.g. language="pt", where the model still picks up spoken Finnish words just fine. Sometimes I even see it "accidentally" translating something into Portuguese (things that between Finnish and Portuguese are in no way phonetically related but the model picks up the correct meaning in other language).

The intermediate layers of multilingual models certainly should be finding out commonalities among languages and providing a representation that makes such intermixing possible. Further research is required ;-)

[jongwook]

Thanks for sharing the experience. As you mentioned, it's possible that the language detection might be focusing too much on the phonetics and less on the linguistic content, making it less robust to accents.

Whisper supports X->X transcription and X->English translation, but not translation to non-English language. Setting the language tag different from what's in the audio is out-of-distribution, since that setting is not in the training data except some noisy data resulted from misclassification by the language classifier. So it's not surprising that the model does its best while switching between Finnish and Portuguese outputs.

It'd be great to have the ability to translate between any languages and to better handle accents and code-switching speech. Lots of further research avenues indeed!