README.md

SelfDZSR++ and SelfTZSR+ (TPAMI 2024)

Official PyTorch implementation of SelfDZSR++ and SelfTZSR++

This work is extended from SelfDZSR (ECCV 2022, GitHub).

Self-Supervised Learning for Real-World Super-Resolution from Dual and Multiple Zoomed Observations
IEEE TPAMI, 2024
Zhilu Zhang, Ruohao Wang, Hongzhi Zhang, Wangmeng Zuo
Harbin Institute of Technology, China

  |  

1. Improvements Compared to SelfDZSR

1.1 Improvements in Methodology

• Introduce patch-based optical flow alignment to further mitigate the effect of the misalignment between data pairs. (Section 3.2.1: Patch-based Optical Flow Alignment)
• Present LOSW (Local Overlapped Sliced Wasserstein) loss to generate visually pleasing results. (Section 3.5: LOSW Loss and Learning Objective)
• Extend DZSR (Super-Resolution based on Dual Zoomed Observations) to multiple zoomed observations, where we present a progressive fusion scheme for better restoration. (Section 3.6: Extension to Multiple Zoomed Observations)

1.2 Improvements in Experiments

• Evaluate the proposed method on a larger iPhone camera dataset. (Section 4.2: Quantitative and Qualitative Results)
• Evaluate the extended method that is based on triple zoomed images. (Section 4.2: Quantitative and Qualitative Results)
• Compare #FLOPs of different methods. (Section 4.3: Comparison of #Parameters and #FLOPs)
• More ablation experiments, including (1) effect of different loss terms (Section 5.3: Effect of LOSW Loss); (2) effect of different reference images and fusion schemes (Section 5.4: Effect of Different Refs and Fusion Schemes); (3) effect of scaling up models (Section 5.5: Effect of Scaling up Models).

2. Preparation and Datasets

2.1 Prerequisites

• Python 3.x and PyTorch 1.12.
• OpenCV, NumPy, Pillow, tqdm, lpips, scikit-image and tensorboardX.

2.2 Dataset

• Nikon camera images can be downloaded from this link.
• iPhone camera images can be downloaded from this link.

3. Quick Start

3.1 Pre-Trained Models

• All pre-trained models are provided in this link. Please place ckpt folder under SelfDZSR_PlusPlus folder.

• For simplifying the training process, we provide the pre-trained models of feature extractors and auxiliary-LR generator. The models for Nikon and iPhone camera images are put in the ./ckpt/nikon_pretrain_models/ and ./ckpt/iphone_pretrain_models/ folder, respectively.

• For direct testing, we provide the 8 pre-trained DZSR and TZSR models (dzsr_nikon_l1, dzsr_nikon_l1sw, dzsr_iphone_l1, dzsr_iphone_l1sw, tzsr_nikon_l1, tzsr_nikon_l1sw, tzsr_iphone_l1, and tzsr_iphone_l1sw) in the ./ckpt/ folder. Taking tzsr_nikon_l1sw as an example, it represents the TZSR model trained on the Nikon camera images using $l_1$ and local overlapped sliced Wasserstein loss terms.

3.2 Training

• Modify dataroot, camera, data, model, and name
• Run sh train.sh

3.3 Testing

• Modify dataroot, camera, data, model, and name
• Run sh test.sh

3.4 Note

• You can specify which GPU to use by --gpu_ids, e.g., --gpu_ids 0,1, --gpu_ids 3, --gpu_ids -1 (for CPU mode). In the default setting, all GPUs are used.
• You can refer to options for more arguments.

4. Results

5. Citation

If you find it useful in your research, please consider citing:

@inproceedings{SelfDZSR,
    title={Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations},
    author={Zhang, Zhilu and Wang, Ruohao and Zhang, Hongzhi and Chen, Yunjin and Zuo, Wangmeng},
    booktitle={European Conference on Computer Vision (ECCV)},
    year={2022}
}

@article{SelfDZSR_PlusPlus,
    title={Self-Supervised Learning for Real-World Super-Resolution from Dual and Multiple Zoomed Observations},
    author={Zhang, Zhilu and Wang, Ruohao and Zhang, Hongzhi and Zuo, Wangmeng},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
    year={2024},
    publisher={IEEE}
}

6. Acknowledgement

This repo is built upon the framework of CycleGAN, and we borrow some code from C2-Matching and DCSR, thanks for their excellent work!