G3FA: Geometry-Guided GAN for Face Animation (BMVC 2024)

This is the official repository for G3FA: Geometry-Guided GAN for Face Animation, presented at BMVC 2024.

Overview

Animating a human face image from a single source frame involves generating a natural-looking, identity-consistent representation that mimics the movements of a driving video. While Generative Adversarial Networks (GANs) have demonstrated promising results in real-time face reenactment, they often lack the geometric consistency provided by graphics-based methods.

G3FA bridges this gap by integrating 3D geometric information derived from 2D images into a GAN-based face animation framework. Our method utilizes inverse rendering techniques to extract 3D facial geometry properties, enhancing the generator’s output through a weighted ensemble of discriminators. By combining 2D motion warping with volumetric rendering, G3FA captures intricate motion dynamics, producing high-quality, geometrically consistent animations.

Key Features

• Incorporates 3D facial geometry using inverse rendering.
• Leverages a weighted ensemble of discriminators for enhanced feedback.
• Combines 2D motion warping and volumetric rendering for detailed motion dynamics.
• Achieves state-of-the-art performance on VoxCeleb2 and TalkingHead benchmarks.

paper

Getting Started

Prerequisites

You will need Python 3.8 or later.

Installation

1. Clone the Repository

   git clone https://github.com/dfki-av/G3FA.git
   cd G3FA

2. Set Up the Conda Environment

   conda create -n g3fa_env python=3.10 -y
   conda activate g3fa_env

3. Install Required Packages

   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
   pip install -r requirements.txt

4. Download Pre-Trained Checkpoints Checkpoint

Running the Demo

To run the live demo, run the following command:

python live_demo.py --source_image path/to/source.png \
               --checkpoint checkpoints/g3fa.pt \

To-Do

• Including training scripts
• Adding evaluation scripts

Acknowledgements

This repository builds upon the following works:

• First Order Motion Model for Image Animation
• FNeVR

Citation

@inproceedings{Javanmardi_2024_BMVC,
  author    = {Alireza Javanmardi and Alain Pagani and Didier Stricker},
  title     = {G3FA: Geometry-guided GAN for Face Animation},
  booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
  publisher = {BMVA},
  year      = {2024},
  url       = {https://papers.bmvc2024.org/0657.pdf}
}