How different are the newly released "Playground" models that have SDXL architecture but were trained from the ground up? Work in A111 but do not merge.

[CCpt5]

A new group released an alternative txt2img model that's basically SDXL structure, but it was trained completely on its own. They've released a few models of it on huggingface:

https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic
https://huggingface.co/playgroundai/playground-v2-1024px-base
https://huggingface.co/playgroundai/playground-v2-512px-base
https://huggingface.co/playgroundai/playground-v2-256px-base

I'm not familiar with the expert functions of model mixer (and/or Supermerger etc) like mbw, so I'm curious if anyone knows if there could be a way to merge those with normal SDXL 1.0 models and fine tunes? Perhaps they just have a different header? According to a social media post they trained it in part of a lot of Midjourney images which could make it pretty cool to blend w/ SDXL models.

Thanks for any insight!!

Model Description
Developed by: [Playground](https://playground.com/)
Model type: Diffusion-based text-to-image generative model
License: [Playground v2 Community License](https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic/blob/main/LICENSE.md)
Summary: This model generates images based on text prompts. It is a Latent Diffusion Model that uses two fixed, pre-trained text encoders ([OpenCLIP-ViT/G](https://github.com/mlfoundations/open_clip) and [CLIP-ViT/L](https://github.com/openai/CLIP/tree/main)). It follows the same architecture as [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0).

[wkpark]

from https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic#using-the-model-with-automatic1111comfyui

I guess you already know,

Using the model with Automatic1111/ComfyUI

In order to use the model with software such as Automatic1111 or ComfyUI you can use playground-v2.fp16.safetensors file.

it means that there is another safetenors checkpoint model compatible with SDXL already. so we can try merging them and see the results.

[CCpt5]

Yea, I have the safetensor (non-diffuser) version, but it doesn't seem to be -exactly- the same format as Stabilitie's models. A1111 does generate images from it natively though....so it must be close?

Merging gives the color noise image. Your extension doesn't recognize it as SDXL - warns that it's an SDv1 model. The "Model Toolkit" extension calls it damaged/unknown and junk. Maybe there's a way w/ just merging parts over that have the important data - I'm haven't learned how to go that deep w/ merging.....

From model toolkit (snippet):

Statistics
Total keys: 2125 (10.33 GB), Unknown keys: 1928 (9.88 GB).

Architecture
CLIP-v1-BROKEN
CLIP-v1
CLIP-v1-SD
Rejected
UNET-v1-SD: Missing required keys (444 of 686)
model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.norm3.weight [640]
model.diffusion_model.output_blocks.7.1.transformer_blocks.0.norm3.bias [640]
model.diffusion_model.output_blocks.4.0.out_layers.0.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.ff.net.2.bias [640]
...
UNET-v1-Inpainting: Missing required keys (445 of 686)
model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.norm3.weight [640]
model.diffusion_model.output_blocks.7.1.transformer_blocks.0.norm3.bias [640]
model.diffusion_model.output_blocks.4.0.out_layers.0.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.ff.net.2.bias [640]
...
UNET-v1-Pix2Pix: Missing required keys (445 of 686)
model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.norm3.weight [640]
model.diffusion_model.output_blocks.7.1.transformer_blocks.0.norm3.bias [640]
model.diffusion_model.output_blocks.4.0.out_layers.0.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.ff.net.2.bias [640]
...
UNET-v2-SD: Missing required keys (434 of 686)
model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.norm3.weight [640]
model.diffusion_model.output_blocks.7.1.transformer_blocks.0.norm3.bias [640]
model.diffusion_model.output_blocks.4.0.out_layers.0.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.ff.net.2.bias [640]
...
UNET-v2-Inpainting: Missing required keys (435 of 686)
model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.norm3.weight [640]
model.diffusion_model.output_blocks.7.1.transformer_blocks.0.norm3.bias [640]
model.diffusion_model.output_blocks.4.0.out_layers.0.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.ff.net.2.bias [640]
...
UNET-v2-Depth: Missing required keys (435 of 686)
model.diffusion_model.output_blocks.4.1.transformer_blocks.0.norm1.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.norm3.weight [640]
model.diffusion_model.output_blocks.7.1.transformer_blocks.0.norm3.bias [640]
model.diffusion_model.output_blocks.4.0.out_layers.0.bias [1280]
model.diffusion_model.output_blocks.8.1.transformer_blocks.0.ff.net.2.bias [640]
...
UNET-XL-SD: Missing required keys (624 of 1680)
model.diffusion_model.output_blocks.1.1.transformer_blocks.0.norm3.weight [1280]
model.diffusion_model.output_blocks.2.1.transformer_blocks.0.norm3.bias [1280]
model.diffusion_model.output_blocks.1.1.transformer_blocks.7.attn1.to_k.weight [1280,1280]
model.diffusion_model.output_blocks.0.1.transformer_blocks.3.ff.net.2.weight [1280,5120]
model.diffusion_model.output_blocks.0.1.transformer_blocks.1.attn2.to_k.weight [1280,2048]
...
UNET-XL-Refiner: Missing required keys (1219 of 1220)
model.diffusion_model.input_blocks.4.1.transformer_blocks.3.norm3.bias [768]
model.diffusion_model.output_blocks.3.1.transformer_blocks.1.norm2.weight [1536]
model.diffusion_model.input_blocks.7.1.transformer_blocks.0.norm3.weight [1536]
model.diffusion_model.input_blocks.4.1.transformer_blocks.1.ff.net.0.proj.weight [6144,768]
model.diffusion_model.output_blocks.9.0.skip_connection.weight [384,1152,1,1]
...
VAE-v1-SD: Missing required keys (16 of 248)
first_stage_model.encoder.mid.attn_1.k.weight [512,512,1,1]
first_stage_model.encoder.mid.attn_1.q.bias [512]
first_stage_model.encoder.mid.attn_1.q.weight [512,512,1,1]
first_stage_model.decoder.mid.attn_1.proj_out.bias [512]
first_stage_model.encoder.mid.attn_1.v.weight [512,512,1,1]
...
SD-v1: Missing required classes
VAE-v1
UNET-v1
SD-v1-Pix2Pix: Missing required classes
VAE-v1
UNET-v1-Pix2Pix
SD-v1-ControlNet: Missing required classes
VAE-v1
ControlNet-v1
UNET-v1
Unknown
first_stage_model.decoder.mid.attn_1.norm.weight [512]
first_stage_model.decoder.mid.attn_1.to_k.bias [512]
first_stage_model.decoder.mid.attn_1.to_k.weight [512,512]
first_stage_model.decoder.mid.attn_1.to_out.0.bias [512]
first_stage_model.decoder.mid.attn_1.to_out.0.weight [512,512]
first_stage_model.decoder.mid.attn_1.to_q.bias [512]
first_stage_model.decoder.mid.attn_1.to_q.weight [512,512]
first_stage_model.decoder.mid.attn_1.to_v.bias [512]
first_stage_model.decoder.mid.attn_1.to_v.weight [512,512]
first_stage_model.decoder.mid.block_1.conv1.bias [512]
first_stage_model.decoder.mid.block_1.conv1.weight [512,512,3,3]
first_stage_model.decoder.mid.block_1.conv2.bias [512]
first_stage_model.decoder.mid.block_1.conv2.weight [512,512,3,3]
first_stage_model.decoder.mid.block_1.norm1.bias [512]
first_stage_model.decoder.mid.block_1.norm1.weight [512]
first_stage_model.decoder.mid.block_1.norm2.bias [512]
first_stage_model.decoder.mid.block_1.norm2.weight [512]
first_stage_model.decoder.mid.block_2.conv1.bias [512]
first_stage_model.decoder.mid.block_2.conv1.weight [512,512,3,3]
first_stage_model.decoder.mid.block_2.conv2.bias [512]
first_stage_model.decoder.mid.block_2.conv2.weight [512,512,3,3]
first_stage_model.decoder.mid.block_2.norm1.bias [512]
first_stage_model.decoder.mid.block_2.norm1.weight [512]
first_stage_model.decoder.mid.block_2.norm2.bias [512]
first_stage_model.decoder.mid.block_2.norm2.weight [512]
first_stage_model.decoder.norm_out.bias [128]
first_stage_model.decoder.norm_out.weight [128]
first_stage_model.decoder.up.0.block.0.conv1.bias [128]
first_stage_model.decoder.up.0.block.0.conv1.weight [128,256,3,3]
first_stage_model.decoder.up.0.block.0.conv2.bias [128]
first_stage_model.decoder.up.0.block.0.conv2.weight [128,128,3,3]
first_stage_model.decoder.up.0.block.0.nin_shortcut.bias [128]
first_stage_model.decoder.up.0.block.0.nin_shortcut.weight [128,256,1,1]
first_stage_model.decoder.up.0.block.0.norm1.bias [256]
first_stage_model.decoder.up.0.block.0.norm1.weight [256]
first_stage_model.decoder.up.0.block.0.norm2.bias [128]