[1.1.446] Effective region mask for ControlNet/IPAdapter

[huchenlei]

Previously mask upload was only used as an alternative way for user to specify more precise mask. It was only supported for inpaint and ipadapter CLIP mask. See answer to #2793.

However, this feature seems to be under-used. We observe that SD Forge uses the mask upload UI to specify effective region. (Note: I am not calling it attn mask as I feel it misleading to the attn mask in transformer argument). In 1.1.446, we implemented the UI to allow people use the UI the same way forge does.

Useage example:
Previously there is no way to prevent depth controlnet to not affect part of the image. If you use pure black to fill the depth map, the result will not be satisfying.

However with effective region mask, now you can limit the ControlNet effect to certain part of image. Here we are only allowing depth controlnet to control left part of image.

There will be a more user friendly region planner tool later to help layout control region for different controlnet unit. #2823. Please stay tuned.

[a3nima]

This is really cool. Any tips on how to use it with IP Adapter Face ID? Seems to work kind of, but not as well as with only 1 controlnet.

[huangjianyi0701]

Hello, I got an error during the use of mask,
the error is as follows:

The environment is as follows:
sd:1.8
controlnet:1.1.455
torch:2.4.1
cuda:12.2

[huangjianyi0701]

[huangjianyi0701]

I found that your method works when the diagram is born, when the mask is redrawn, the mask uploaded by ipadapter will not take effect and report an error, he will enter this function three times, and the inconsistency of the dimensions of the out and mask at the third output will result in an error, what is the problem please?