Does the general learning rate matter when we're doing LORA?

[nicolae-stroncea]

When we're doing LORA there seem to be 3 parameters:

• Learning Rate
• LORA Learning Rate
• Lora Text Learning Rate

Is the learning rate being used as well or is it just set to the LORA learning rate when it's being run

[sailxjx]

wiki ELI5-Training says that the learning rate is better starting from 2e-6. For LoRa, LR defaults to UNET 1e-4, Text is 5e-5, which is the parameter in the original paper of lora.

I tried 2e-6 in training, basically it takes 300-500 epochs to get more similar faces, a bit slow, but the result is very good.

And I also tried 6e-6, it takes 100-200 epochs to fit the training set, but I feel the quality is slightly worse (psychologically).

I never tried 1e-4 or 5e-5, I think the result may be bad, because when I use lr as 6e-6, the generated sample image after 300 epoch is corrupted, so I don't think bigger the lr will be better.

[double-fire-0]

the learning rate 2e-6 you mentation is the learning rate of lora or other items?

[sailxjx]

I keey both Lora UNET and Lora Text Encoder learning rate to 6e-6

[double-fire-0]

got it! thanks

[klubkelli]

I have played around with the learning rate a lot and it does matter and there is no set number that works better than others. It really depends not only how many images you are using when training the lora but also the quality of the images.