Loss not reducing

[JRMeyer]

>>> dalonlobo
[January 16, 2018, 6:15am]

Hi

I'm training the Deepspeech model with the following parameter, on an
audio dataset of size 72hrs:27min:34sec split between train:dev:test in
the ratio 60:20:20. slash
But the loss does not seam to reduce after 4 epochs, don't know what is
the problem, any help is much appreciated.

Parameters

> #!/bin/sh
> set -xe
> if [ ! -f DeepSpeech.py ]; then
> echo 'Please make sure you run this from DeepSpeech's top level directory.'
> exit 1
> fi;
> python -u DeepSpeech.py slash
> --train_files /datadrive/dalon/video_datasets/other-train.csv slash
> --dev_files /datadrive/dalon/video_datasets/other-dev.csv slash
> --test_files /datadrive/dalon/video_datasets/other-test.csv slash
> --train_batch_size 16 slash
> --dev_batch_size 8 slash
> --test_batch_size 8 slash
> --validation_step 1 slash
> --display_step 5 slash
> --limit_train 0 slash
> --limit_dev 0 slash
> --limit_test 0 slash
> --n_hidden 2048 slash
> --epoch 50 slash
> --checkpoint_dir /datadrive/dalon/video_datasets/checkpoint/ slash
> --export_dir /datadrive/dalon/video_datasets/model_export/ slash
> --decoder_library_path /datadrive/dalon/DeepSpeech/native_client/libctc_decoder_with_kenlm.so slash
> --alphabet_config_path /datadrive/dalon/pre_trained/models/alphabet.txt slash
> --lm_binary_path /datadrive/dalon/lm_models/ds_full_lm_o5.binary slash
> --lm_trie_path /datadrive/dalon/lm_models/ds_full_lm_trie slash
> --fulltrace True slash
> --use_warpctc True slash
> --early_stop False slash
> --report_count 100 slash
> --use_seq_length False slash
> '$'

Output slash
![Screenshot from 2018-01-16

[This is an archived TTS discussion thread from discourse.mozilla.org/t/loss-not-reducing]

[JRMeyer]

>>> lissyx
[January 16, 2018, 9:59am]

With only 72 hours, if you take 60% of it for train set, you have 43
hours. We use n_hidden=2048 with 100 times that amount of audio, so I
guess you should reduce that parameter. Try much lower values, and do
binary search to find appropriate width :).

[Archived Post]

[JRMeyer]

>>> dalonlobo
[January 16, 2018, 10:13am]

thanks, I'll try reducing the
hidden layers

[Archived Post]

[JRMeyer]

>>> kdavis
[January 16, 2018, 11:02am]

I'd also suggest adding the following hyperparameter values

...
--dropout_rate 0.30 slash
--default_stddev 0.046875 slash
--learning_rate 0.0001 slash
...

If you plan on changing n_hidden you should also adjust
default_stddev in accordingly

[Archived Post]

[JRMeyer]

>>> george_fedoseev
[April 12, 2018, 10:03am]

Could you please tell how
default_stddev should depend on n_hidden?

[Archived Post]

[JRMeyer]

>>> kdavis
[April 12, 2018, 11:00am]

There are various schools of thought on this.

However, generally what I've been guided by is Xavier
initialization slash [1 slash ].

[Archived Post]