Tiled upscale

[monstruosoft]

I've reworked the tiled upscale code and decided to open this discussion before opening a new PR. Tiled upscale has been around since SD came out as it's the simplest way to use SD to upscale an image. I know this implementation is not perfect but it's meant to allow users running FastSD CPU in CPU alone on low-end machines, like my own, to upscale and improve their SD results.
I can barely generate images with SD Turbo and LCM models at 512x512 resolution; at that resolution SD tends to work great for closeup images but not so great otherwise. As an example, an SD Turbo generation for "knight girl, on a forest road":

And after applying tiled upscale:

Note that the face in this example required the generation of an extra tile for the face alone, think of it as "poor man's inpainting".

[rupeshs]

@monstruosoft Nice, Did you remove the black borders from the image?

[monstruosoft]

Yes, the black borders are removed and it now works as expected, as far as I can tell; I solved it by generating tiles slightly larger than 512x512, I thought SD was constrained to the default resolutions (512, 768, 1024) but I realized that it can generate images with arbitrary sizes as well.
I've also added the option to pass a JSON file with custom upscale settings as an argument to the CLI version; the idea is that, eventually, the same code and functions for tiled upscale could be used from the GUI versions by dynamically building a dictionary object containing those options.

[rupeshs]

@monstruosoft cool,but I'm wondering this arbitrary size breaks anything? Feel free to create PR. Great work. I have plan to add edsr for upscale.

[monstruosoft]

I didn't change any of the image generation code; as before, I don't want to mess with it and risk breaking something. I just set the image width and height in config.lcm_diffusion_setting to the desired resolution and SD will generate an image with that size, seems to work without any issues.
Yes, I realized that you're working on adding EDSR upscaling, I think a combination of both upscaling methods would give the best results.
Thanks, will create a PR.

[rupeshs]

OpenVINO SD upscale

Upscaled image

[monstruosoft]

Oh, you got the OpenVINO issue solved, that's great!.
Thanks!.

I've noticed that the upscaled images tend to turn out kinda blurry and less detailed than the original; unfortunately, I have no idea how to solve it or improve the output. It seems to work better with some models than with others, though, in particular anime models:

[rupeshs]

@monstruosoft Beta 24 release https://github.com/rupeshs/fastsdcpu/releases/tag/v1.0.0-beta.24

[monstruosoft]

Your work over the past few days has been impressive, it's hard to keep up with the changes!.
Thanks!.

[rupeshs]

Thanks, I just reduced the coupling between components in upscaler workflow, so it will be easier for us to maintain FastSD CPU code.

[monstruosoft]

I've noticed that both Readme.md and the CLI help mention 2x upscale and 512x512 resolution. While it's true that those are the default values, there's more settings you can use with a JSON file or a dynamically created custom_settings dictionary.

{
  "source_file": "/home/monstruosoft/fastsdcpu/results/5cc34c84-fb86-40a6-8eaf-369788d8c6a1-1.png",
  "target_file": null,
  "target_format": "jpg",
  "tile_size": 256,
  "tile_overlap": 16,
  "scale_factor": 2.0,
  "strength": 0.3,
  "tiles": []
}

Let's say you have an image with a resolution of 600x600 pixels, if you use the default tile_size of 256, it'll require 9 tiles to upscale it; if, instead, you set the tile_size to 300, it'll only generate 4 tiles with a size of 600x600 pixels. Depending on whether your PC can handle the generation of images with higher resolutions, you might prefer one tile size over the other.

As you may be wondering, you can also set a different scale_factor; the size of the generated tiles will be tile_size*scale_factor and, again, it depends on your hardware capabilities how high can you go. For example, the image on the left was upscaled with a scale_factor of 3.0, a strength of 0.1 and using a tile_size of 152 (so that the generated output tiles would be near to 512x512, which is what I can generate on my machine):

In the next post I'll try to describe the custom tiles settings.

[monstruosoft]

It's not uncommon for the default upscale tiling to split a face in two and the resulting image may be sometimes acceptable but sometimes it may turn out with different colored eyes or misaligned face features, so you might want to fix it. Sometimes, particularly with full body shots, the face may be missing details after upscale, for example (I intentionally cropped the output image):

So you might define a custom tile for the face using a JSON file for the upscaling option like this:

{
  "source_file": "/home/monstruosoft/fastsdcpu/results/f2e9b52b-b26e-4bfc-a21d-3f8a4f83f4ce-1.png",
  "target_file": "/home/monstruosoft/fastsdcpu/results/f2e9b52b-b26e-4bfc-a21d-3f8a4f83f4ce-1_2x_upscale_1707158056.png",
  "target_format": "jpg",
  "tile_size": 256,
  "tile_overlap": 16,
  "scale_factor": 2.0,
  "strength": 0.6,
  "tiles": [{
    "x": 216,
    "y": 131,
    "w": 80,
    "h": 67,
    "scale_factor": 6.0,
    "mask_box": null,
    "prompt": null
  }]
}

This JSON file defines a single tile for the face alone; source_file is the original unscaled image, target_file is the upscaled image with missing face details, the top-level scale_factor defines the scale between the original and the upscaled files, typically 2x. The tiles array defines a tile at the corresponding position and size on the source image.

Note that it's likely that using image variations with low denoising strength to generate the new tile using the original face region won't fix the face so you have to use a higher denoising strength, 0.6 in the code above. Also note that the tile can have a different scale factor, in this case I chose 6x for SD to generate an image with a resolution close to 512x512 so that it can add more detail. This JSON file generates the following result:

Note that you could also try to use the already upscaled image both as the source and target as follows (note that the top-level scale_factor is set to 1.0, while the new tile scale_factor is set to 4.0):

{
  "source_file": "/home/monstruosoft/fastsdcpu/results/f2e9b52b-b26e-4bfc-a21d-3f8a4f83f4ce-1_2x_upscale_1707158056.png",
  "target_file": "/home/monstruosoft/fastsdcpu/results/f2e9b52b-b26e-4bfc-a21d-3f8a4f83f4ce-1_2x_upscale_1707158056.png",
  "target_format": "jpg",
  "tile_size": 256,
  "tile_overlap": 16,
  "scale_factor": 1.0,
  "strength": 0.6,
  "tiles": [{
    "x": 446,
    "y": 264,
    "w": 128,
    "h": 128,
    "scale_factor": 4.0,
    "mask_box": null,
    "prompt": null
  }]
}

In cases like this, where there's too little information in the original image for SD to work with, you can improve the tile creation with a prompt; if you add a prompt, the code will use img2img instead of the default image variations mode to generate the tiles:

{
  "source_file": "/home/monstruosoft/fastsdcpu/results/f2e9b52b-b26e-4bfc-a21d-3f8a4f83f4ce-1_2x_upscale_1707158056.png",
  "target_file": "/home/monstruosoft/fastsdcpu/results/f2e9b52b-b26e-4bfc-a21d-3f8a4f83f4ce-1_2x_upscale_1707158056.png",
  "target_format": "jpg",
  "tile_size": 256,
  "tile_overlap": 16,
  "scale_factor": 1.0,
  "strength": 0.6,
  "tiles": [{
    "x": 446,
    "y": 264,
    "w": 128,
    "h": 128,
    "scale_factor": 4.0,
    "mask_box": null,
    "prompt": "1girl, face closeup, beautiful, blonde, red lips, smiling"
  }]
}

So, as you can see, there's a lot of combinations to define custom tiles and there's no magic formula that will work right in every case, it'll typically require some trial and error.

Finally, you might notice that at higher denoising strength the tile seams are more visible, this could be improved by using a custom mask instead of the default mask box. Note that custom masks are not currently supported but should be trivial to add.

The target_format setting is meant to allow you to save images in JPG format. If you're like me and are always running out of space in your hard disk, it might be a good option; however, this setting is not currently honored and images are saved by default in PNG format.

[rupeshs]

@monstruosoft I appreciate your great work.
I just added JPEG support to FastSD CPU (see master branch). We are nearing 1.0.0 release. My main focus is app should be easy to use for everyone.

[monstruosoft]

Thanks for JPEG support; for users like me that are always running out of space in their HDD, this is a good option to have.
I've created a new PR with some minor changes, most important change is that you can now define a top level prompt that might improve tiled upscale tile creation, for example:

{
  "source_file": "/home/monstruosoft/fastsdcpu/results/5cc34c84-fb86-40a6-8eaf-369788d8c6a1-1.png",
  "target_file": null,
  "target_format": "jpeg",
  "tile_size": 256,
  "tile_overlap": 16,
  "prompt": "knight girl, on a forest road",
  "scale_factor": 2.0,
  "strength": 0.3,
  "tiles": []
}

Using this JSON file, the top level prompt will be used by default as the prompt for all tiles.