Please test my new front-end (Seth's AI Tools, open source, Windows)

[SethRobinson]

I've just finished modifying my AI Tools front-end to work with the standard AUTOMATIC1111 stable-diffusion-webui's api! (previously I required my own hacked up version)

Features:

• Live update integration (image and masking) with Photoshop or other image editors
  text to image, inpainting, image interrogation, face fixing, upscaling, tiled texture generation with preview
• Drag and drop images in as well as paste images from the windows clipboard
• Pan/zoom with thousands of images on the screen
• Mask painting with controllable brush size
• Can utilize multiple servers (three video cards on one machine? Run three servers!) allowing seamless use of all remote GPUs for ultra fast generation and a single click to change the active model
• Neat workflow that allows evolving images with loopback while live-selecting the best alternatives to shape the image in real-time
• Open source, uses the Unity game engine and C# to do stuff with AI art
• Privacy respected - does not phone home or collect any statistics, purely local usage
• A few extra features like background removal are possible, but only if my modified server is used, (it autodetects)

Youtube movie showing how it works

If anyone wants to give it a go, here are the links:

Direct download to the Windows client zip (36 MB) (After running aitools_client.exe, click Configuration and change the server URL/port if you aren't running on localhost)

Github page with full source code and more info

Note: You must be running your SD WebUI server with the --api parm, and if it's not local, the --listen parm as well.

Appreciate feedback/suggestions

[Ladypoly]

I like that it is based on unity which has an extremely flexible UI System.

What I don't like: The UI
It always opens on my second screen in full screen, with no other way than alt+ F4 to close it.

For me in the current state it is untestable to give you a real feedback.

[SethRobinson]

Hey thanks for the feedback.

I thought I set it to open in windowed mode at 1920X1080 by default but I might have screwed something up. In Windowed mode, you should be able to drag/resize it as needed.

If it's full screen, try hitting Alt-Enter to get it to go back to Windowed, then you should be able to drag it to the other monitor. I'll do some tests on a fresh computer over here for the next build.

[Norgus]

I think this is the beginnings of something very fun.
A few things I feel might be nice improvements:

• ability to raise the batch generation number (so more get simultaneously generated)
• realtime generation preview
• ability to switch between inpainting at full rez (what it seems to do?) and inpainting the whole image.
• more intuitive buttons (maybe coloured icon style?)
• Some way of viewing prompt history
• some way to copy the prompt from an image on screen or dragged-in from outside

Having the responsive interface and fast generation is great :)

[SethRobinson]

Hey thanks for the suggestions!

ability to switch between inpainting at full rez (what it seems to do?) and inpainting the whole image.

Yes, all inpainting is full rez - if you're dealing with a large image, you can drag the rect around to where you want to work on and keep the rectangle size at 512X512 or 768X768 as needed for the best results on a smaller area. Maybe for rects larger than 768 (for example, covering the entire image) it should automatically use other modes, hrm.

some way to copy the prompt from an image on screen or dragged-in from outside

Not sure if I understand correctly, but if you click the little "I" button above an image, it will perform interrogation on the image and change the "prompt" box with new text. (how accurate/useful the prompt guess is may be debatable though...)

[Norgus]

The inpainting doesn't behave as I would have expected - if you use 'inpaint at full resolution' in the interface, auto1111 cuts down the image to a bounding box around the masked area that fits your aspect ratio, does the img2img on that at your specified resolution, and then puts the result back into the overall image for the results. If you are giving the user of your UI the ability to drag a custom bounding box, I would assume you'd want to be cutting the bounding box out, scaling it to the img2img resolution, getting an img2img on it with the inpainting mask, then scaling the result back for the result image in your interface.

I can't tell for sure, but it feels to me like the bounding box I drag in your UI doesn't end up as the context given to the inpainting process.

it will perform interrogation on the image
this is handy, but what I'm referring to is the image metadata that is optionally (on by default) saved to the images generated by the automatic1111 interface. It allows you to bring in an image and it's prompt + settings on the webui via the png info tab, so I was suggesting some parallel functionality to that.

While experimenting again this time to write this reply, I think I discovered a problem in loading models that are in subdirectories?
e.g. all of my version 2.0 and 2.1 models are in 'SD_Automatic111\models\Stable-diffusion\2' and simply won't load when selected (although they load if selected from the good old webui) - I have some 1.x series models in other subdirectories and those won't load either, but all of my root directory models will load correctly.

[SethRobinson]

About the inpainting rect:

My processing is only done on the rectangle selected - that rect is cut out, processed on its own, then "put back" into the image. (SD won't see any part of the image not in the rect, so prompts need to keep that in mind) Everything is 1 to 1 pixel, no scaling/distortion can happen. However, if you use a rect size SD doesn't like (64x64 etc) you'll just get garbage back so basically it's only going to give good results if you use a 512 or 768 sized rect.

Subdir model bug:

Hmm, I put a model in a "models\Stable-diffusion\2" directory and it was able to be selected/loaded ok - but my server is running on linux, maybe yours is on Windows? I suspect it's a bug in the SD WebUI API itself, will need to do a test with a Windows server and verify that.

[Norgus]

I will add one more serious problem that needs remedying actually:

this is when not even generating and my PC fans rage like a storm.

[SethRobinson]

Yikes, that's bad. I've released V0.51 that hopefully has this issue fixed. (Mine dropped from 11% to 2% after this change (rtx 3090))

Same download link as before. Thanks!

[Norgus]

It definitely made a big difference on the resource usage! hovers around 20% GPU idle but goes down to less than 10% when generating (I guess adaptive clock speed might account for that)

edit: oh yeah, and just a couple of % for CPU use, so muuuch better :)

[cooperdk]

VERY interesting, will try out! :-)

[cooperdk]

Error when running. I have API enabled and am running from localhost.
Seems it looks for a dir/file inside the webui directory named aitools\get_info.json

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\anyio\streams\memory.py", line 94, in receive
    return self.receive_nowait()
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\anyio\streams\memory.py", line 89, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\base.py", line 77, in call_next
    message = await recv_stream.receive()
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\anyio\streams\memory.py", line 114, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\fastapi\applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\base.py", line 106, in __call__
    response = await self.dispatch_func(request, call_next)
  File "Q:\StableDiffusion\webui\extensions\auto-sd-paint-ext\backend\app.py", line 358, in app_encryption_middleware
    res: StreamingResponse = await call_next(req)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\base.py", line 80, in call_next
    raise app_exc
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\base.py", line 69, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 24, in __call__
    await responder(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 43, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\routing.py", line 69, in app
    await response(scope, receive, send)
  File "Q:\StableDiffusion\webui\venv\lib\site-packages\starlette\responses.py", line 334, in __call__
    raise RuntimeError(f"File at path {self.path} does not exist.")
RuntimeError: File at path Q:\StableDiffusion\webui\aitools\get_info.json does not exist.

[SethRobinson]

You can safely ignore that - it's normal. For some reason the AUTOMATIC1111 WebUI server doesn't simply give a 404 when a URL is asked for that doesn't exist but instead generates that exception. Hopefully this can be changed in the future. (this isn't something I can change on my end)

My client asks for this URL from the server once during initialization to see if it has extra extensions, if this file is missing/error/404 is returned, it knows it can't.

[cooperdk]

You can safely ignore that - it's normal. For some reason the AUTOMATIC1111 WebUI server doesn't simply give a 404 when a URL is asked for that doesn't exist but instead generates that exception. Hopefully this can be changed in the future. (this isn't something I can change on my end)

My client asks for this URL from the server once during initialization to see if it has extra extensions, if this file is missing/error/404 is returned, it knows it can't.

But it stops completely and a message "Error" is displayed in every image box in your client.
I mean, nothing happens at all.

[SethRobinson]

Hmm, I just did a fresh install of stable-diffusion-webui on my local Windows 11 and it did work for me, strange. We might be running it differently? (I typed "webui.bat --api" in a dos prompt from the stable-diffusion-webui dir)

My output looks identical to yours, except it does work for me:

Hmm, got an idea. Can you please try this beta version? It removes the .json check, so there will be no server error.

Beta version download link

[cooperdk]

I'll get back to you later on this. 😊

[cooperdk]

There is no message in the CLI now.

But this is what I see in your UI:

The messages propagate almost instantaneously when I press "generate".

Also, my webui-user.bat does include both the --api and the --listen arguments.

If I'm really fast, I have time to see this error 422 at the bottom:

Since the web ui doesn't seem to have any kind of debugging ability, I am at a loss as to why this happens. Your UI does communicate with it but for some reason, the ui cannot process whatever it sends.

BTW My web ui runs in a Windows Terminal with admin privileges.

[cooperdk]

Thru ngrok, I got errors from the API from your client which I have registered as an issue in your own repository instead of here.

SethRobinson/aitools_client#1

[y0himba]

Seth, this is brilliant. First SD UI to work for me right "out of the box" with no issues. I am addicted. I am watching your project page for releases.

Suggestions: Change batch sizes, and a UI for the configuration as a lot of users won't know how to copy the path to their favorite editor and so forth.

If I am missing where to change the amount of images generated and the batch size, please let me know.

[cooperdk]

Suggestions: Change batch sizes, and a UI for the configuration as a lot of users won't know how to copy the path to their favorite editor and so forth.

I'd say if people don't know how to copy a path into a text field, they shouldn't even attempt to mess with Stable Diffusion...
Still, not a bad idea. Should be in a config area instead :-)

[SethRobinson]

Hey y0himba, currently there is no way to specify batch generation.

In my tests I found that batch generation was not that much faster (like 5.5 seconds to create two images, compared to 3 seconds to create one) and I preferred to see them pop up quicker, one by one - allowing more interactive prompt/loopback target changes. (no need to wait for a batch)

The system is designed to "generate non-stop in the background" and when any specific thing is requested (upscaling, inpaint, etc) it internally schedules it to happen ASAP and then goes back to generating more images. This system works nicely with multiple servers/gpus.

(After clicking "Generate", it just generates images non-stop with all connected servers until it's clicked again to stop it - same behavior with the inpainting when the little "!" button is used on an image)

Maybe I'm wrong and batch generation saves more time than I thought? It would be possible to add.

[y0himba]

I likéto generate in small batches, revise, another small batch, revise, to me at least it seems easier, and faster to set the number of images to generate and it stops automatically.

If I want a large amount of images and I need to let them generate while I am away from the keyboard, this limits the amout that will generate and then it stops, unattended.

Thank you for considering it!

[cooperdk]

I think it would be nice to limit generation to x rows or x images too. Let the user decide how to use it.

Also, consider my suggestion with random tag removal, I think that will increase the usability of your client a lot, and it's relatively easy to add. (Could just be a checkbox and a piece of coffee to select a word or sentence separated with comma) to remove.

[SethRobinson]

Thanks, I've released V0.54 with those two features added (click the gear icon next to the Generate button to see them)

[SethRobinson]

Thanks everybody - I've released V0.53 with some bugfixes.

Same link to download the latest version

• BUGFIX in 0.51: GUI no longer eats way too many GPU cycles (Norgus)
• BUGFIX in 0.52: Inpainting mode no longer always selects "original" regardless of dropdown setting
• BUGFIX in 0.53: Should work and not give errors when Windows regional settings are not set to US (cooperdk)
• BUGFIX in 0.53: Fixed issue where switching models wouldn't work from subdirs on servers running on Windows (Norgus, cooperdk)

[cooperdk]

There is another issue with the new subdirs fix.

Your UI converts Windows directory delimiters () to Unix-style forward slashes, which is not possible in Windows by default, causing the UI to fallback to the first model it finds it can load. I think Python should support it, but it doesn't work.

[SethRobinson]

Dang it! Thanks, will try to fix for the next update.

Planning to throw in a generation limit and the random tag removal too, btw I didn't understand this: "Could just be a checkbox and a piece of coffee to select a word or sentence separated with comma".

Piece of coffee? I'm thinking maybe you meant piece of code? (to randomly select one item between commas to remove)

[cooperdk]

LOL, yeah, that's what I meant and that's what sometimes happens when I write in English on my mobile phone. I use swype for typing and sometimes I hit a wrong char and don't always notice the error.

Sounds good. BTW, safetensors are passed from the server api and work fine, too. But the beauty of api is that you can easily implement good functionality in a visually nice package.

Future? I see your UI maybe being able to convert images with depth maps into meshes or something like that. And outpainting.

[cooperdk]

BTW the loading of models issue can be circumvented until your next update by loading the model first in your UI and then in the Automatic1111 web ui. That way, your model will use the correct model. (It might not even be necessary to select it in your UI as long as it's loaded in the web ui).

[SethRobinson]

Yeah this is correct, it really doesn't matter if the client knows what the model is (the only thing it affects is the default generation X/Y size, it will change to 768 if it notices 768 in the filename).

Sometimes I set all three servers connected to different models (using the webUI) - kind of fun to simultaneously see a result of SD 1.X, 2.X and anime from the same prompt.

[cooperdk]

It looks and works great.
Sorry I have been AFK a bit, just pushed myself into ordering a new computer that better fits the 3060, so I got an i7-13700K with really fast RAM and a new 3500MB/s NVMe drive, liquid cooling and sh#§.

(Honestly, bought primarily for 3D stuff)

[NoCrypt]

I still can't believe one of my favourite game's creator is making stable-diffusion front end tools. Great job!

[y0himba]

Wait...what?!

[jtac]

Growtopia ;)