New Feature: "ZOOM ENHANCE" automatically fixes small details like faces and hands!

[ThereforeGames]

I'm pleased to announce the latest addition to the Unprompted extension: it's the [zoom_enhance] shortcode.

Named after the totally-not-fake technology from CSI, zoom_enhance allows you to automatically upscale small details within your image where Stable Diffusion tends to struggle. It is particularly good at fixing faces and hands in long-distance shots.

How does it work?

The [zoom_enhance] shortcode searches your image for specified target(s), crops out the matching regions and processes them through [img2img]. It then blends the result back into your original image. All of this happens behind-the-scenes without adding any unnecessary steps to your workflow. Just set it and forget it.

Features and Benefits

• Great in both txt2img and img2img modes.
• The shortcode is powered by the [txt2mask] implementation of clipseg, which means you can search for literally anything as a replacement target, and you get access to the full suite of [txt2mask] settings, such as padding and negative_mask.
• It's also pretty good at deepfakes. Set mask="face" and replacement="another person's face" and check out the results.
• It applies a gaussian blur to the boundaries of the upscaled image which helps it blend seamlessly with the original.
• It is equipped with Dynamic Denoising Strength which is based on a simple idea: the smaller your replacement target, the worse it probably looks. Think about it: when you generate a character who's far away from the camera, their face is often a complete mess. So, the shortcode will use a high denoising strength for small objects and a low strength for larger ones.
• It is significantly faster than Hires. Fix and won't mess up the rest of your image.
• Compatible with A1111's color correction setting, which you'll probably want to use to avoid issues related to over-saturation.
• In many cases, it makes the "restore faces" option obsolete. Try the shortcode with and without "restore faces" and see for yourself.
• Unlike "restore faces," [zoom_enhance] won't interfere with the style of your image. Face restoration is biased toward photography. With this shortcode, you can provide a subject like "illustration of walter white face" to avoid that problem.
• Compatible with all models. You can even use [set sd_model] to change your checkpoint just during the upscale step.
• Compatible with batch size and batch count.

More Examples

Don't take my word for it. Judge for yourself.

How to Use

You can access the GUI through Unprompted » Wizard » Shortcodes » zoom_enhance:

Or slam this into your prompt:

[after]{zoom_enhance}[/after]

It goes inside of an [after] block because it is supposed to execute after the generation of an image.

By default, it will look for face and replace it with an upscaled face. If you're making a specific person --such as Walter White--you should provide a more specific replacement value like so:

[after]{zoom_enhance replacement="walter white face"}[/after]

If you want to fix hands instead of a face, you can try something like this:

[after]{zoom_enhance mask="fingers" replacement="closeup hand" max_denoising_strength=0.9 precision=120}[/after]

Note: it's going to take some trial and error to find optimized settings for hands. Let me know if you find a config that works better than the one above.

You can place multiple zoom_enhance blocks back-to-back. Fixes multiple problem areas in one go.

Limitations

• Because this shortcode calls an img2img task in an unusual manner, it may not be compatible with every extension. Try disabling your other extensions if you run into issues.
• This shortcode has not yet been throughly battle-tested. Your bug reports are appreciated.

Bonus - send me some prompts in the comments and I'll be happy to show you some more before-after examples!

[HWiese1980]

Cool idea, dude!

Unfortunately, I'm seeing the following error:

Traceback (most recent call last):
  File "K:\stablediffusionwebui\extensions\unprompted\lib_unprompted\shortcodes.py", line 117, in render
    return str(self.handler(self.token.keyword, self.pargs, self.kwargs, context))
  File "K:\stablediffusionwebui\extensions\unprompted\lib_unprompted\shared.py", line 61, in handler
    return(self.shortcode_objects[f"{keyword}"].run_atomic(pargs, kwargs, context))
  File "K:\stablediffusionwebui\extensions\unprompted/shortcodes\stable_diffusion\zoom_enhance.py", line 55, in run_atomic
    else: all_images = self.Unprompted.shortcode_user_vars["init_images"]
KeyError: 'init_images'

Got any idea what's causing this error? Anything I forgot to do maybe?

Extension and A1111 should already be most recent version. I updated everything just minutes ago.

[ThereforeGames]

Hi @HWiese1980,

Are you using the shortcode inside of an [after] block? Make sure your prompt is in the following format (or use the GUI wizard if you prefer):

photo of walter white[after]{zoom_enhance}[/after]

If that's not it, then perhaps it's a compatibility issue with some other extension. Try disabling other extensions to see if it makes a difference.

[HWiese1980]

My bad, obviously. I didn't use it that way. That solves this initial problem.

For some reason though, I don't seem to get it working. I added the shortcode like that, but for some reason it does not seem to do anything at all. It is enabled, the checkbox is checked.

[ThereforeGames]

Hi @HWiese1980, try updating the extension to v7.7.2 and enable the new "Use Workaround" setting:

It seems to fix the issue for those running Python 3.10 (I'm a bit behind, developing on 3.9!)

[Rainz3]

Installed but am receiving this error

`Downloading clipseg model weights...
Error running postprocess: I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\scripts\unprompted.py
Traceback (most recent call last):
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\lib_unprompted\shortcodes.py", line 117, in render
return str(self.handler(self.token.keyword, self.pargs, self.kwargs, context))
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\lib_unprompted\shared.py", line 61, in handler
return(self.shortcode_objects[f"{keyword}"].run_atomic(pargs, kwargs, context))
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted/shortcodes\stable_diffusion\zoom_enhance.py", line 74, in run_atomic
mask_image = self.Unprompted.shortcode_objects["txt2mask"].run_block(set_pargs,set_kwargs,None,target_mask)
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted/shortcodes\stable_diffusion\txt2mask.py", line 211, in run_block
self.image_mask = get_mask().resize((self.init_image.width,self.init_image.height))
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted/shortcodes\stable_diffusion\txt2mask.py", line 115, in get_mask
model.load_state_dict(torch.load(d64_file), strict=False);
File "I:\Stable\stable-diffusion-webui-directml\modules\safe.py", line 106, in load
return load_with_extra(filename, extra_handler=global_extra_handler, *args, **kwargs)
File "I:\Stable\stable-diffusion-webui-directml\modules\safe.py", line 151, in load_with_extra
return unsafe_torch_load(filename, *args, **kwargs)
File "I:\Stable\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 789, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "I:\Stable\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 1131, in _load
result = unpickler.load()
File "I:\Stable\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 1101, in persistent_load
load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
File "I:\Stable\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 1083, in load_tensor
wrap_storage=restore_location(storage, location),
File "I:\Stable\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 215, in default_restore_location
result = fn(storage, location)
File "I:\Stable\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 182, in _cuda_deserialize
device = validate_cuda_device(location)
File "I:\Stable\stable-diffusion-webui-directml\venv\lib\site-packages\torch\serialization.py", line 166, in validate_cuda_device
raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "I:\Stable\stable-diffusion-webui-directml\modules\scripts.py", line 404, in postprocess
script.postprocess(p, processed, *script_args)
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\scripts\unprompted.py", line 455, in postprocess Unprompted.shortcode_objects[i].after(p,processed)
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted/shortcodes\basic\after.py", line 32, in after
self.Unprompted.process_string(self.Unprompted.parse_alt_tags(content,"after"))
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\lib_unprompted\shared.py", line 157, in parse_alt_tags
return(self.shortcode_parser.parse(string,context))
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\lib_unprompted\shortcodes.py", line 219, in parse
return stack.pop().render(context)
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\lib_unprompted\shortcodes.py", line 58, in render
return ''.join(child.render(context) for child in self.children)
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\lib_unprompted\shortcodes.py", line 58, in
return ''.join(child.render(context) for child in self.children)
File "I:\Stable\stable-diffusion-webui-directml\extensions\unprompted\lib_unprompted\shortcodes.py", line 121, in render
raise ShortcodeRenderingError(msg) from ex
lib_unprompted.shortcodes.ShortcodeRenderingError: An exception was raised while rendering the 'zoom_enhance' shortcode in line 1.`

I see it mention CUDA so would it be due to having an ATI?

[ThereforeGames]

Hi @Rainz3,

This should be fixed in v7.8.0. Give it a try and let me know if the problem persists. Thank you!

[ThereforeGames]

Hi all,

Zoom Enhance now supports multiple subjects. See this post for more info:

https://www.reddit.com/r/StableDiffusion/comments/11qsfcv/update_zoom_enhance_now_supports_multiple_subjects/

[MrLawbreaker]

Does this not work for batch Img2Img? I am trying to convert a bunch of pictures but the shortcode seems to only be applied on the first picture of the batch.

Only 01000 gets 2 pictures (Workaround adds one, add original image and another enhanced one)

[skyler14]

Does this not work for batch Img2Img? I am trying to convert a bunch of pictures but the shortcode seems to only be applied on the first picture of the batch. Only 01000 gets 2 pictures (Workaround adds one, add original image and another enhanced one)

I too would like to know about using this in batch processing.

Also I was wondering if zoom_enhance or other shortcodes work on textual inversions?

[skyler14]

I second the question on if textual inversions work for this?

[Shoot2Skoot]

How would I achieve this:

Compatible with all models. You can even use [set sd_model] to change your checkpoint just during the upscale step.

I've tried sometihng like this:

photo of thing[after]{zoom_enhance replacement="{{set sd_model}}model_name{{/set}}face"}[/after]

Does the model name need to include the ".ckpt"? What about if the model is in a subfolder?

[jonas-klesen]

Anyone can give an example on how to use the set sd_model in zoom_enhance?

[NinjaTech]

It would be great if someone created something like this WITHOUT the need to install unprompted.

[TheCodeSlinger]

So I just wrote my own. Introducing FacePop. https://github.com/TheCodeSlinger/FacePop