Cannot get the validation loss to decrease on text detection

[dvivarelli]

Hello again.
I'm trying to fine tune the detection phase.
The training set is 50 images with 30 annotations on each image
Val set is 7 images with 30 annotations on each image

I start the training using the command:

train_pytorch.py ./train_data ./val_data db_mobilenet_v3_large  -j 1 --batch_size 1 --wb --epochs 200 --lr 0.0001  --pretrained

but after a few epochs val loss increases

How can I avoid the overfitting?

[fg-mindee]

Hi @dvivarelli 👋

There are few things in place here:

• you are using a batch size of 1 which won't be pretty on the batch norm layers, I would strongly recommend increasing that
• the smaller the dataset, and the more epochs you do, the higher the probability of overfitting seriously. In your case 50 images is considered pretty extreme for deep learning training (your model is very capable of memorizing all your training samples).
• however you are starting from pretrained params: so you are starting from a model that has a very good generalization capacity. Have you tried to check if the pretrained version is not already enough? (most likely here you are degrading the perf with this training setup)
• if you absolutely want to finetune the model, I suggest two things:
  • getting more labeled data & using a decent batch size
  • freezing backbone and training only the head (by passing --freeze-backbone in your command)

Also, if you want to check the good LR to pick here, I suggest trying out the LR finder option available on all our training scripts (cf. #703). That should help you pick a good LR!

Hope all of this helps :)