cyclical learning rate

[joeismad]

Added a cyclical learning rate option for training. In case anyone is interested, just putting this out there.

Example of CLR in yaml:
training:
lr_cyclical: true
lr_cyclical_steps: 5000
lr_cyclical_max: .2
lr_cyclical_min: .02

[ASilver]

Nice one, though it is worth mentioning that one could easily use CLR before. I guess the question is how is this managed? Just peak LR at the midway point and then back to the minimum? There are a lot of approaches to CLR with differing rates of increase and decrease, such as more steps to reach the peak, followed by a quicker fall.