API: Script order on alwayson_scripts matters

[Hugo-Matias]

I'm currently implementing several always on scripts, like ControlNet and Dynamic Prompts, using the newly added api parameter to pass in the required arguments. It's been working quite well just by following the def process() method signature of the script in question to define what arguments to send in the payload and in what order.

Today I found a little peculiarity with this when trying to make MultiDiffusion's Tiled Diffusion work with Dynamic Prompts. Consider this payload example. The relevant part to notice here is the order in which Dynamic Prompts and Tiled Diffusion objects are sent. This payload will not trigger Dynamic Prompts, confirmed by the console log and the result itself.

{
  "enable_hr": false,
  "hr_scale": 0,
  "hr_second_pass_steps": 0,
  "hr_resize_x": 0,
  "hr_resize_y": 0,
  "denoising_strength": 0.52,
  "prompt": "red ferrari",
  "seed": -1,
  "batch_size": 4,
  "n_iter": 1,
  "steps": 20,
  "cfg_scale": 9.5,
  "width": 512,
  "height": 512,
  "restore_faces": true,
  "tiling": false,
  "negative_prompt": "lowres",
  "sampler_index": "DPM\u002B\u002B 2M Karras",
  "alwayson_scripts": {
    "Dynamic Prompts v2.8.4": {
      "args": [
        true,
        false,
        1,
        true,
        true,
        false,
        1.1,
        1.5,
        100,
        1,
        false,
        false,
        true,
        false,
        false,
        0,
        "Gustavosta/MagicPrompt-Stable-Diffusion",
        null
      ]
    },
    "Tiled Diffusion": {
      "args": [
        true,
        "MultiDiffusion",
        true,
        true,
        1024,
        1024,
        96,
        96,
        48,
        1,
        "None",
        2,
        false,
        false,
        1,
        []
      ]
    }
  }
}

This, however, will work just fine and reflect the same behavior from WebUI:

"alwayson_scripts": {
   "Tiled Diffusion": { ... },
   "Dynamic Prompts v2.8.4": { ... }

@Vespinian Have you noticed this during the development of the implementation? I'm really not sure why this happens and what's the root but I'm guessing there is something going on between the two scripts and how they are handled by the WebUI internally. In a remote case it might be something wrong with my installation.

In the end, this is not a problem by any means, I'm just spreading awareness that this might occur in other situations with other scripts save you some debug time.

EDIT: It seems like it might be the Tiled Diffusion script acting up, either by improper argument list passed in (the script still works) or perhaps it does something internally that messes the rest of the argument list when called from the api. Running it on it's own makes the cutoff.py script complain about missing parameters:

TypeError: Script.process() missing 2 required positional arguments: 'intp' and 'debug'

This shouldn't happen because there is no reference to the Cutoff script in the payload, so it shouldn't be enabled by default and generating with the same parameters from the web ui doesn't exhibit this issue...Very weird behavior. Not sure if I should proceed discussion here or the script's own repo.

Perhaps the solution to this is to explicitly send the arguments to all the always on scripts installed setting the ones we don't want to use to false.

[Vespinian]

Interesting, I'll have to take a look at the extensions code. If you want to, you can try out PR #8669. It should be stop scripts that you don't want to run from running but you will have to specify all alwayson_scripts that you want to run, even if they don't take args. One caveat though is that it can make some errors if the script is for some reason playing with the global ScriptRunners instead of the one passed in the processing object.

There is also a more experimental change on one of the branches in my fork that does the changes above and arg list resizing but depending on what is happening I don't know if it's going to fix whatever Tiled Diffusion is doing.

[Vespinian]

So my quick findings about the code of Tiled Diffusion:
You are going to need a ton more args then what you posted. When I tried it and printed self.args_from and self.args_to it went from index 1 to index 88 aka 87 args.

What I think I am seeing is the first 15 you posted that are in the function signature, then I think the 16th one *bbox_control_states is the 9 args of a bbox_control [e, x, y ,w, h, prompt, neg_prompt, blend_mode, feather_ratio] times whatever BBOX_MAX_NUM is. Which would makes sense since the default value of BBOX_MAX_NUM for me was 8, so 15 + 9*8 = 87

So, I haven't tried this, but the args for Tiled Diffusion as of the day and time of this post with a BBOX_MAX_NUM of 8 (which seems to come from a setting in shared.cmd_opts.md_max_regions) it would look something like this. Of course, you will need to change the var names I put there to actual values.

"Tiled Diffusion": {
      "args": [
        true, "MultiDiffusion", true, true, 1024, 1024, 96, 96, 48, 1,"None", 2, false, false, 1,
        e0, x0, y0 ,w0, h0, prompt0, neg_prompt0, blend_mode0, feather_ratio0,
        e1, x1, y1 ,w1, h1, prompt1, neg_prompt1, blend_mode1, feather_ratio1,
        e2, x2, y2 ,w2, h2, prompt2, neg_prompt2, blend_mode2, feather_ratio2,
        e3, x3, y3 ,w3, h3, prompt3, neg_prompt3, blend_mode3, feather_ratio3,
        e4, x4, y4 ,w4, h4, prompt4, neg_prompt4, blend_mode4, feather_ratio4,
        e5, x5, y5 ,w5, h5, prompt5, neg_prompt5, blend_mode5, feather_ratio5,
        e6, x6, y6 ,w6, h6, prompt6, neg_prompt6, blend_mode6, feather_ratio6,
        e7, x7, y7 ,w7, h7, prompt7, neg_prompt7, blend_mode7, feather_ratio7
      ]
    }

You can probably get away with this for all 8 bbox for a quick test. That should make them all disabled.
false, 0, 0, 0, 0, "", "", "Background", 0

Not quite sure why would cutoff complain or Dynamic Prompt not run though. For cutoff, with the PR I posted, that problem should go away if it's not included. For Dynamic Prompt it's probably error related, if Tiled diffusion throws errors because it's getting Nones then Dynamic Prompt may not run.

[Hugo-Matias]

Thanks for looking into this in such depth! I suspected that might have something to do with the boxed regions, I haven't integrated them into my app yet and was under the assumption that sending an empty array or null would be sufficient to fill the *bbox_control_states arg. Now I can better appreciate the work put into ControlNet's hijack methods in order to send an arbitrarily sized list of units.

Regarding Cutoff, it seems to happen only in txt2img for some reason, also after the latest MultiDiffusion commits the missing arguments increased to 5, it's quite odd that it's trying to consume arguments from somewhere but you're probably right about the silent errors being the cause for this.

Since this is not a breaking issue and passing the expected arguments for the bbox controls as you suggested seems to solve the problems for both extensions, I'll wait for the merge for now but that certainly sounds like a nice feature to prevent this kind of behavior in the future.

[SkillfulPainter]

Hi, it seems that I cannot use Region Prompt Control using the code above. I think it may because the version of Multidiffusion I use is different. What version are you guys using? Can you provide a link or something?