Unlocking Voices, Unleashing Possibilities: Your Words, Our Recognition!

Code:

Below are the step to setup the code and perform training

Setup:

After setting up the code as below, update the paths appropriately

git clone https://github.com/ksasi/asr.git

cd asr

git clone https://github.com/speechbrain/speechbrain.git

Install Dependencies:

cd asr/speechbrain

pip install -r requirements.txt

cd ..

pip install -r requirements.txt

• copy all the files from <root_path>/code to <root_path>/speechbrain/recipes/LibriSpeech/ASR/transformer

Datasets :

• Create a directory named datasets under asr
• Download [LibriSpeech] (https://www.openslr.org/12) (specifically train-clean-100, test-clean and dev-clean partitions)

Execute the below steps to generate Noisy LibriSpeech dataset

cd code

nohup python generate_noisy_librispeech.py --wham_dir <root_path>/datasets/wham_noise --libri_dir <root_path>/datasets/LibriSpeech >> <root_path>/logs/librispeech_wham_noise.out &

Models Evaluation (Using Noisy LibriSpeech)

Evaluation of pretrained wav2vec2 and Conformer (pretrained on original Librispeech)

• wav2vec2 : speechbrain/asr-wav2vec2-librispeech
• Conformer : speechbrain/asr-conformer-transformerlm-librispeech

cd code

nohup python model_eval.py --libri_dir <root_path>/datasets/LibriSpeech/test-clean >> <root_path>/logs/eval_wav2vec2_conformer.log &

Evaluation of pretrained Branchformer (pretrained on original Librispeech)

• branchformer : pyf98/librispeech_100_ctc_e_branchformer

cd code

nohup python branchformer_eval.py --libri_dir <root_path>/datasets/LibriSpeech/test-clean >> <root_path>/logs/eval_branchformer.log &

Fine-tune Conformer (on Noisy LibriSpeech)

cd <root_path>/speechbrain/recipes/LibriSpeech/ASR/transformer

nohup python finetune_conformer.py <root_path>/hparams/conformer_large.yaml --data_folder=<root_path>/datasets/LibriSpeech >> <root_path>/logs/finetune_conformer.out &

Fine-tune TSConformer (on Noisy LibriSpeech)

• TSConformer is a Custom Conformer Architecture using TaylorSeries Linear Attention

cd <root_path>/speechbrain/recipes/LibriSpeech/ASR/transformer

nohup python finetune_tsconformer.py <root_path>/hparams/tsconformer_large.yaml --data_folder=<root_path>/datasets/LibriSpeech >> <root_path>/logs/finetune_tsconformer.out &

Complexity for Conformer Vs TSConformer

cd code

python maccounter_asr.py >> <root_path>/logs/flopcount.out

Demo

Demo of Automatic Speech Recognition on Noisy Data can be executed by running ASR_Demo.ipynb ipython notebook in the demo folder

References

• Speechbrain - Github Link
• EER Metric - blog
• SepFormer Huggingface - Link
• Torchmetrics - Link
• branchformer - Link
• wav2vec2 - Link
• Conformer - Link
• Taylor Series Linear Attention - Link