Learning Rate Scheduler

⚠️Warning⚠️

This page exist for the sole purpose of explaining some of the settings.
Take every Learning Rate Scheduler suggestion with a grain of salt, especially the more creative ones.
When it's usually discussed, it's borderline faith.

Thou shalt always use a constant learning rate, for it is the path to wisdom.

Warmup ♨️

Warmup is a solution to early over-fitting. By slowly ramping up the learning rate, there is more time for the model to adapt to the new data. This should in turn, result in better training.

Constant

Probably the safest option, and the easiest to understand. When trying new datasets, this should probably be your go to. Learning rate will remain constant