(SOLVED - Roll back NVIDIA Driver to 531.79) Just recently updated after a couple weeks of not using Dreambooth. It is suddenly DRAMATICALLY slower.

[morphinapg]

I've gone from about 3-4 it/s average, to about 3 s/it, with all of the same settings. That's 10x slower! What in the world has changed recently to cause this? And what command can I use to go back to before whatever those changes were? Looks like the last model I made was on May 28th, but I can't be sure when the last update I did was. I'm using the main branch.

[morphinapg]

Okay, I just figured out how to roll back to what I had before (using the answer here) but my performance is still really bad, although slightly better than before. Could it be a bad driver update, perhaps? Or maybe something A1111 did?

EDIT: reverting A1111 didn't fix it

[morphinapg]

Just saw these posts:

https://www.reddit.com/r/StableDiffusion/comments/141is23/performance_regression_with_latest_nvidia_drivers/

https://www.reddit.com/r/StableDiffusion/comments/1458fv4/nvidia_driver_memory_changes/

Seems like that may be related

If it's an issue of NVIDIA being too aggressive swapping VRAM to RAM, then perhaps I need to lower batch size back to 1 instead of the 2 I usually used just fine in older drivers.

[morphinapg]

Getting considerably better performance with batch size 1, although I'm getting about 2 it/s, so it's still a loss in performance compared to the 3-4 it/s I would get with Batch Size 2, even though I increase epochs by 40% when I do that. That means with the new driver and batch size 1, it will take about 25% longer to train a model than it used to. I really hope they fix this.

[morphinapg]

I was able to solve the problem by rolling back to the 531.79 Nvidia Driver. I was getting over 4 it/s with a batch size of 2 with this. I highly recommend anybody with an NVIDIA card to roll back to 531.79 for now, until NVIDIA fixes this