How to decide the appearance seed?

[Aoiryo]

Hi, thanks for you amazing work!
I have some doubts on the concept, seed, mentioned in your paper.
Considering using an existing image as the input, how can I decide the appearance seed?
Also, is the appearance seed simply a random integer, which is a hyper-parameter for the user to adjust? If so, what do you suggest to find the best value?

Thanks in advance!

[iamNCJ]

Hi, thanks for your interest in our work.

The reason why we need an appearance seed is that although the main content is defined by the provisional image, there is still huge ambiguity on appearance details and the original lighting (e.g. how specular is the material? Is a certain part bright because of the bright lighting in the provisional image or just because the texture itself is bright?). Therefore, DiLightNet is modeling the distribution of all possible local shading results, and the appearance seed determines which possible result is sampled during the diffusion process.

In the implementation, the appearance seed corresponds to the sequence of random noise used in the diffusion process in stage 2. it's an integer number used as the seed of the random noise generator.

DiLightNet/demo/relighting_gen.py

Lines 55 to 58 in e7758bb

	generator = torch.Generator(device=device).manual_seed(seed)
	images = pipe(
	prompt, num_inference_steps=steps, generator=generator, image=hint, num_images_per_prompt=num_imgs_per_prompt, guidance_scale=cfg, output_type='np',
	).images # [N, H, W, C]

And in our provided inference script, the default appearance seed is 3407 and you can set your own with --seed. You can check here for a detailed description of the arguments.

I usually generate 4 images at a time with different seeds, they all follow the global lighting effects but have diversity in local appearance details, and I would pick the one I like the most as the final one. It can be more of a personal preference and just as any diffusion model, you might need to try different seeds and prompts to get the result you like the most.