StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation

📝 This repository contains the official PyTorch implementation of the following paper:

StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation
Wonjong Jang, Gwangjin Ju, Yucheol Jung, Jiaolong Yang, Xin Tong, Seungyong Lee, SIGGRAPH 2021

🚀 >> Project page
🚀 >> Fast-forward video

Overview

Explanation

The key component of our method is shape exaggeration blocks that are used for modulating coarse layer feature maps of StyleGAN to produce desirable caricature shape exaggerations. We first build a layer-mixed StyleGAN for photo-to-caricature style conversion by swapping fine layers of the StyleGAN for photos to the corresponding layers of the StyleGAN trained to generate caricatures. Given an input photo, the layer-mixed model produces detailed color stylization for a caricature but without shape exaggerations. We then append shape exaggeration blocks to the coarse layers of the layer-mixed model and train the blocks to create shape exaggerations while preserving the characteristic appearances of the input.

Requirements

✔️ PyTorch 1.3.1
✔️ torchvision 0.4.2
✔️ CUDA 10.1/10.2
✔️ dlib 19.22.0
✔️ requests 2.23.0
✔️ tqdm 4.46.2

⚠️ If you are using Anaconda environment and get errors regarding compiler version mismatch, check issue #1.

Usage

First download pre-trained model weights:

bash ./download.sh

Train

python -m torch.distributed.launch --nproc_per_node=N_GPU train.py --name EXPERIMENT_NAME --freeze_D

Test

Test on user's input images:

python test.py --ckpt CHECKPOINT_PATH --input_dir INPUT_IMAGE_PATH --output_dir OUTPUT_CARICATURE_PATH --invert_images

We provide some sample images. Test on sample images:

python test.py --ckpt CHECKPOINT_PATH --input_dir examples/samples --output_dir examples/results --invert_images

It inverts latent codes from input photos and generates caricatures from latent codes.

Examples

Input image	Output caricature

Citation

If you find this code useful, please consider citing:

@article{Jang2021StyleCari,
  author    = {Wonjong Jang and Gwangjin Ju and Yucheol Jung and Jiaolong Yang and Xin Tong and Seungyong Lee},
  title     = {StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation},
  booktitle = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH)},
  publisher = {ACM},
  volume = {40},
  number = {4},
  year = {2021}
}

Download pre-trained models

🏷️ StyleCariGAN
🏷️ Photo-StyleGAN (generator_ffhq.pt)
🏷️ Caricature-StyleGAN (generator_cari.pt)
🏷️ Photo-Attribute-Classifier (photo_resnet.pth)
🏷️ Cari-Attribute-Classifier (cari_resnet.pth)

Contact

📫 You can have contact with wonjong@postech.ac.kr or ycjung@postech.ac.kr

License

This software is being made available under the terms in the LICENSE file.

Any exemptions to these terms require a license from the Pohang University of Science and Technology.

Credits

❤️ Our code is based on the official StyleGAN2 implementation and rosinality's StyleGAN2-pytorch code
❤️ Specially thanks to CJWBW who ported our project to Replicate.