finetune get worse result

[DENGBOYU-REX]

Hi, thanks for your great work! I am trying to fine-tune the big-lama model on my custom dataset, but the output after fine-tuning seems worse than the original version.

Here is the original version output:
image:

output:

mask:

And here is the fine-tuned output:
image:

output:

mask:

I fine-tuned the model at 512x512 resolution. Could this be the issue? And i just use perceptual loss from ResNet50 and MSE loss. I would appreciate any advice you can provide!