How do we train lora for Flux-Fill-Dev?

[matabear-wyx]

The newly released Flux-Fill-Dev (Inpainting/Outpainting) is incredible. However, the LoRA I previously trained on Flux-Dev doesn't work directly with Flux-Fill-Dev, so I'm planning to retrain a LoRA specifically for Flux-Fill-Dev.

From my quick analysis, Flux-Fill-Dev takes an input of a 16-channel image latent, a 16-channel masked image latent, and a 64-channel mask. My initial idea for input preparation is to use the 16-channel image latent repeated twice, along with a 64-channel mask filled entirely with zeros (black mask).

Any thoughts or suggestions?

[chengyou-jia]

any new update for your training? i am also trying to train flux-fill-dev with lora.

[matabear-wyx]

Yes, you can try my method, it works properly

[chengyou-jia]

Thank you for your reply! How is the performance of the fill-model trained with LoRA? Would it be possible for you to share the code for the update? I would greatly appreciate it! 😊

[matabear-wyx]

What I did is really simple and raw, as I discussed:
`def pack_fill_latents(latents, batch_size, num_channels_latents, height, width):
latents = latents.view(
batch_size, num_channels_latents, height // 2, 2, width // 2, 2
)
latents = latents.permute(0, 2, 4, 1, 3, 5)
latents = latents.reshape(
batch_size, (height // 2) * (width // 2), num_channels_latents * 4
)
# Step 2: Repeat the packed latents
repeated_latents = latents.repeat(1, 1, 2) # Repeat along the channel dimension

# Step 3: Create a black mask of the required shape
mask = torch.zeros((batch_size, (height // 2) * (width // 2), 8 * 8 * 4), device=latents.device)

# Step 4: Concatenate repeated latents with the black mask
packed_fill_latents = torch.cat([repeated_latents, mask], dim=-1)

return packed_fill_latents`

[chengyou-jia]

thanks, so did you prepare a dataset(using real masks)and test it's effectiveness?