Error in training crnn_vgg16_bn on custom data

[adesgautam]

Hi,

I am trying to train crnn_vgg16_bn on custom dataset on colab using tensorflow.
Chars in my labels are:
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ`abcdefghijklmnopqrstuvwxyz

Running the training using this:

python references/recognition/train_tensorflow.py final_data/train_set final_data/val_set crnn_vgg16_bn --pretrained --epochs 50 

I am getting the following error when using the default vocab:

Validation set loaded in 0.001157s (99 samples in 2 batches)
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
Traceback (most recent call last):
  File "references/recognition/train_tensorflow.py", line 260, in <module>
    main(args)
  File "references/recognition/train_tensorflow.py", line 103, in main
    vocab=VOCABS[args.vocab]
  File "/content/doctr/doctr/models/recognition/crnn/tensorflow.py", line 242, in crnn_vgg16_bn
    return _crnn('crnn_vgg16_bn', pretrained, **kwargs)
  File "/content/doctr/doctr/models/recognition/crnn/tensorflow.py", line 219, in _crnn
    load_pretrained_params(model, _cfg['url'])
  File "/content/doctr/doctr/models/utils/tensorflow.py", line 55, in load_pretrained_params
    model.load_weights(f"{params_path}{os.sep}{internal_name}")
  File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 1171, in assert_is_compatible_with
    raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (127,) and (124,) are incompatible
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-0._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-0._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-1._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-1._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-2._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-2._inbound_nodes
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details

And when I am using the latin vocab:
The command:

python references/recognition/train_tensorflow.py final_data/train_set final_data/val_set crnn_vgg16_bn --pretrained --epochs 50 --vocab latin 

And the error:

Namespace(amp=False, arch='crnn_vgg16_bn', batch_size=64, epochs=50, input_size=32, lr=0.001, name=None, pretrained=True, resume=None, show_samples=False, test_only=False, train_path='final_data/train_set', val_path='final_data/val_set', vocab='latin', wb=False, workers=None)
Validation set loaded in 0.0009081s (99 samples in 2 batches)
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
DEBUG:tensorflow:Layer lstm_1 will use cuDNN kernels when running on GPU.
Traceback (most recent call last):
  File "references/recognition/train_tensorflow.py", line 260, in <module>
    main(args)
  File "references/recognition/train_tensorflow.py", line 103, in main
    vocab=VOCABS[args.vocab]
  File "/content/doctr/doctr/models/recognition/crnn/tensorflow.py", line 242, in crnn_vgg16_bn
    return _crnn('crnn_vgg16_bn', pretrained, **kwargs)
  File "/content/doctr/doctr/models/recognition/crnn/tensorflow.py", line 219, in _crnn
    load_pretrained_params(model, _cfg['url'])
  File "/content/doctr/doctr/models/utils/tensorflow.py", line 55, in load_pretrained_params
    model.load_weights(f"{params_path}{os.sep}{internal_name}")
  File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 1171, in assert_is_compatible_with
    raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (95,) and (124,) are incompatible
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-0._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-0._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-1._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-1._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-2._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).decoder.layer_with_weights-2._inbound_nodes
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

But when i am not using the pretrained argument the training works fine.
Why is it so ? And can I use the pretrained model to finetune on my dataset ?

[charlesmindee]

Hi @adesgautam,

When you use a pretrained recognition model, the model is built with vocab used for training, i.e the one in the cfg of the model. For the crnn_vgg16_bn, it is french_legacy, our old french vocab with 124 chars. In the training script, the default vocab is the french one which has 127 chars. So the model tries to match 2 different shapes. What you need to do is pass the --vocab 'legacy_french' arg to the training script if you want to use the pretrained version, otherwise if you want to specify your own vocabulary you need to start from scratch, otherwise the tensor shapes won't match!
I hope I am clear, let me know.

Have a nice day!

[adesgautam]

Hi @charlesmindee

Thank you so much for replying so quick and the detailed explanation 🙌
Using legacy_french it's training fine now.

Also, after training how do I use it for inference ? I checked the discussions but it was mentioned for pytorch models.
Is there a link for that ?

[adesgautam]

1 more last thing, what is this printing during training ?
<IPython.core.display.HTML object>

[charlesmindee]

After training you have the best checkpoints that are saved, you can just import the model from doctr, load the checkpoints using the tf load_weightst() method , and after that you can do whatever you want with your trained model. If you want to use it as a recognition predictor (including preprocessing), you need to instantiate a recognition predictor from the architecture you trained, specify the arg pretrained=True and pass the local url to your save checkpoints in the model cfg.

[charlesmindee]

The printing during training, I don't know but it seems that you are trying to display something, you may want to use plt.show() but it doesn't seem to come from doctr.

[adesgautam]

I am not printing anything but anyways I'll check that.

Thanks, I'll try it to load the model.

Have a great day 🙌