Skip to content

Latest commit

 

History

History
73 lines (60 loc) · 3.52 KB

README.md

File metadata and controls

73 lines (60 loc) · 3.52 KB
Raw

SCGN

Implementation of SCGN in paper "Deep View Synthesis via Self-Consistent Generative Networks" [arxiv].

We propose a novel end-to-end deep generative model, called self-consistent generative network (SCGN), to synthesize novel views from given input views relying on image content only.

Requirements

  • Python == 3.x
  • OpenCV-Python
  • Tensorflow >= 1.12
  • Platform: Linux

Usage

1. Clone

git clone https://github.com/zhuomanliu/SCGN.git
cd SCGN/

2. Data Preparation

2.1. Multi-PIE

  • Download dataset from here.
  • According to the Content page, we collect data from varied poses (take 05_1 as the target pose) with the highest illumination (i.e., the 7th illumination).
  • Specifically, the original image would be center-cropped and resized into 224x224.

2.2. KITTI

  • Download the odometry data (color) from here.
  • Collect the first 11 sequences from /dataset/sequences/${seq_id}/image_2/, where seq_id is from 00 to 10, and place the sequences in the folder ./datasets/kitti/data. Similarly, the raw image would be center-cropped and resized into 224x224.
  • Execute ./datasets/kitti/gen_list.py for train-test split and random sampling, then train.csv and test.csv will be generated in ./datasets/kitti/split.
  • Alternatively, we also provide the prepared KITTI dataset. [Google Drive] [Baidu Cloud, pwd: 57gq]

3. Train & Test

  • Download pre-trained models [Multi-PIE, KITTI] and unzip them in the folder ./ckpts for testing.
  • Start training or testing as follows:
# train
./train.sh 0 multipie run-multipie  # for Multi-PIE
./train.sh 0 kitti run-kitti  # for KITTI

# test
./test.sh 0 multipie run-multipie res-multipie  # for Multi-PIE 
./test.sh 0 kitti run-kitti res-kitti  # for KITTI

4. Demo

  • We have provided some samples in ./demo/multipie and ./demo/kitti for inference.
  • Start inference as follows:
# for Multi-PIE
./demo.sh 0 multipie run-multipie ./demo/multipie 15_input_l.png 15_input_r.png 15_result
./demo.sh 0 multipie run-multipie ./demo/multipie 30_input_l.png 30_input_r.png 30_result
./demo.sh 0 multipie run-multipie ./demo/multipie 45_input_l.png 45_input_r.png 45_result

# for KITTI
./demo.sh 0 kitti run-kitti ./demo/kitti 09_input_l.png 09_input_r.png 09_result
./demo.sh 0 kitti run-kitti ./demo/kitti 10_input_l.png 10_input_r.png 10_result

Citation

Please cite the following paper if this repository helps your research:

 
@article{liu2021deep,
  title={Deep View Synthesis via Self-Consistent Generative Network},
  author={Liu, Zhuoman and Jia, Wei and Yang, Ming and Luo, Peiyao and Guo, Yong and Tan, Mingkui},
  journal={IEEE Transactions on Multimedia},
  year={2021},
  publisher={IEEE}
}