Stable diffusion 3.5 not loading the LORA fine tuned weight

[AIML001]

Hi,

I've fine tuned the SD-3.5-medium model. I can also see the validation image are great. But when I'm trying to load the lora checkpoint and generating image it's quality is just like a base model. It looks like it's not loading the model.

I used this code for loading the weights:-

pipe = StableDiffusion3Pipeline.from_pretrained("stabilityai/stable-diffusion-3.5-medium", torch_dtype=torch.bfloat16, use_safetensors=True)
pipe = pipe.to("cuda")
pipe.load_lora_weights('output/experiments/models', weight_name="pytorch_lora_weights.safetensors")

Config Used:-

{
    "--resume_from_checkpoint": "latest",
    "--data_backend_config": "config/multidatabackend.json",
    "--aspect_bucket_rounding": 2,
    "--seed": 42,
    "--minimum_image_size": 0,
    "--disable_benchmark": false,
    "--output_dir": "output/models",
    "--lora_type": "standard",
    "--lora_rank": 64,
    "--max_train_steps": 1500,
    "--num_train_epochs": 0,
    "--checkpointing_steps": 100,
    "--checkpoints_total_limit": 5,
    "--attention_mechanism": "diffusers",
    "--tracker_project_name": "lora-training",
    "--tracker_run_name": "simpletuner-lora",
    "--report_to": "tensorboard",
    "--model_type": "lora",
    "--pretrained_model_name_or_path": "stabilityai/stable-diffusion-3.5-medium",
    "--model_family": "sd3",
    "--train_batch_size": 1,
    "--gradient_checkpointing": "true",
    "--caption_dropout_probability": 0.05,
    "--resolution_type": "pixel_area",
    "--resolution": 1024,
    "--validation_seed": 42,
    "--validation_steps": 100,
    "--validation_resolution": "1024x1024",
    "--validation_guidance": 5.0,
    "--validation_guidance_rescale": "0.0",
    "--validation_num_inference_steps": "20",
    "--validation_prompt": "a image of abc car in front right pose",
    "--mixed_precision": "bf16",
    "--optimizer": "adamw_bf16",
    "--learning_rate": "8e-5",
    "--lr_scheduler": "polynomial",
    "--lr_warmup_steps": 100,
    "--base_model_precision": "int8-quanto",
    "--validation_torch_compile": "false"
}

[bghira]

it's not an issue with this project unfortunately, the weights are saved and loaded correctly. but you could open an issue with the diffusers project. just make sure you're matching all the same settings. a negative prompt is provided by default, for example.

[bghira]

actually, you could try loading one of the checkpoints instead of the main export at the end. 🤷 maybe there is something wrong with the last saved file.

[AIML001]

@bghira I think it is more of choosing the right parameter. I'm able to get someone closer object now.
But my validation result are very good, how can I get all parameters used during validation and during training? I know the file "helpers>training>validation.py" has that logic

[AIML001]

So I checked it's passing prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds & negative_pooled_prompt_embeds.
How can I get this during test?