Pixel Difference Convolution

This repository contains the PyTorch implementation for "Pixel Difference Networks for Efficient Edge Detection" by Zhuo Su*, Wenzhe Liu*, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen and Li Liu** (* Authors have equal contributions, ** Corresponding author). [arXiv, youtube]

The writing style of this code is based on Dynamic Group Convolution.

If you find something useful from our work, please consider citing our paper.

@ARTICLE{pdc-PAMI2023,
  author={Su, Zhuo and Zhang, Jiehua and Wang, Longguang and Zhang, Hua and Liu, Zhen and Pietikäinen, Matti and Liu, Li},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Lightweight Pixel Difference Networks for Efficient Visual Representation Learning}, 
  year={2023},
  volume={45},
  number={12},
  pages={14956-14974},
  keywords={Image edge detection;Feature extraction;Computational modeling;Biological system modeling;Task analysis;Quantization (signal);Object recognition;Binary neural networks;convolutional neural networks;edge detection;efficient representation learning;facial recognition;image classification},
  doi={10.1109/TPAMI.2023.3300513}
}

🚀 Updates:

• Dec. 29, 2021: Add functions with vanilla conv components in models/ops_theta.py
• Aug. 18, 2021: Load checkpoints in trained_models

Running environment

Training: Pytorch 1.9 with cuda 10.1 and cudnn 7.5 in an Ubuntu 18.04 system
Evaluation: Matlab 2019a

Ealier versions may also work~ :)

Dataset

We use the links in RCF Repository (really thanks for that). The augmented BSDS500, PASCAL VOC, and NYUD datasets can be downloaded with:

wget http://mftp.mmcheng.net/liuyun/rcf/data/HED-BSDS.tar.gz
wget http://mftp.mmcheng.net/liuyun/rcf/data/PASCAL.tar.gz
wget http://mftp.mmcheng.net/liuyun/rcf/data/NYUD.tar.gz

To create BSDS dataset, please follow:

1. create a folder /path/to/BSDS500,
2. extract HED-BSDS.tar.gz to /path/to/BSDS500/HED-BSDS,
3. extract PASCAL.tar.gz to /path/to/BSDS500/PASCAL,
4. if you want to evaluate on BSDS500 val set, the val images can be downloaded from this link, please extract it to /path/to/BSDS500/HED-BSDS/val,
5. cp the *.lst files in data/BSDS500/HED-BSDS to /path/to/BSDS500/HED-BSDS/, cp the *.lst files in data/BSDS500 to /path/to/BSDS500/.

To create NYUD dataset, please follow:

1. create a folder /path/to/NYUD,
2. extract NYUD.tar.gz to /path/to/NYUD,
3. cp the *.lst files in data/NYUD to /path/to/NYUD/.

Training, and Generating edge maps

Here we provide the scripts for training the models appeared in the paper. For example, we refer to the PiDiNet model in Table 5 in the paper as table5_pidinet.

table5_pidinet

# train, the checkpoints will be save in /path/to/table5_pidinet/save_models/ during training
python main.py --model pidinet --config carv4 --sa --dil --resume --iter-size 24 -j 4 --gpu 0 --epochs 20 --lr 0.005 --lr-type multistep --lr-steps 10-16 --wd 1e-4 --savedir /path/to/table5_pidinet --datadir /path/to/BSDS500 --dataset BSDS

# generating edge maps using the original model
python main.py --model pidinet --config carv4 --sa --dil -j 4 --gpu 0 --savedir /path/to/table5_pidinet --datadir /path/to/BSDS500 --dataset BSDS --evaluate /path/to/table5_pidinet/save_models/checkpointxxx.pth

# generating edge maps using the converted model, it should output the same results just like using the original model
# the process will convert pidinet to vanilla cnn, using the saved checkpoint
python main.py --model pidinet_converted --config carv4 --sa --dil -j 4 --gpu 0 --savedir /path/to/table5_pidinet --datadir /path/to/BSDS500 --dataset BSDS --evaluate /path/to/table5_pidinet/save_models/checkpointxxx.pth --evaluate-converted

# test FPS on GPU
python throughput.py --model pidinet_converted --config carv4 --sa --dil -j 1 --gpu 0 --datadir /path/to/BSDS500 --dataset BSDS

It is similar for other models, please see detailed scripts in scripts.sh.

The performance of some of the models are listed below (click the items to download the checkpoints and training logs). FPS metrics are tested on a NVIDIA RTX 2080 Ti, showing slightly faster than that recorded in the paper (you probably get different FPS records in different runs, but they will not vary too much):

Model	ODS	OIS	FPS	Training logs
table5_baseline	0.798	0.816	101	log
table5_pidinet	0.807	0.823	96	log, running log
table5_pidinet-l	0.800	0.815	135	log
table5_pidinet-small	0.798	0.814	161	log
table5_pidinet-small-l	0.793	0.809	225	log
table5_pidinet-tiny	0.789	0.806	182	log
table5_pidinet-tiny-l	0.787	0.804	253	log
table6_pidinet	0.733	0.747	66	log, running_log
table7_pidinet	0.818	0.824	17	log, running_log

Evaluation

The matlab code used for evaluation in our experiments can be downloaded in matlab code for evaluation.

Possible steps:

1. extract the downloaded file to /path/to/edge_eval_matlab,
2. change the first few lines (path settings) in eval_bsds.m, eval_nyud.m, eval_multicue.m for evaluating the three datasets respectively,
3. in a terminal, open Matlab like

matlab -nosplash -nodisplay -nodesktop

# after entering the Matlab environment, 
>>> eval_bsds

4. you could change the number of works in parpool in /path/to/edge_eval_matlab/toolbox.badacost.public/matlab/fevalDistr.m in line 100. The default value is 16.

For evaluating NYUD, following RCF, we increase the localization tolerance from 0.0075 to 0.011. The Matlab code is based on the following links:

• HED Implementation
• Original HED
• Piotr's Structured Forest matlab toolbox
• RCF

PR curves

Please follow plot-edge-pr-curves, files for plotting pr curves of PiDiNet are provided in pidinet_pr_curves.

Generating edge maps for your own images

python main.py --model pidinet_converted --config carv4 --sa --dil -j 4 --gpu 0 --savedir /path/to/savedir --datadir /path/to/custom_images --dataset Custom --evaluate /path/to/table5_pidinet/save_models/checkpointxxx.pth --evaluate-converted

The results of our model look like this. The top image is the messy office table, the bottom image is the peaceful Saimaa lake in southeast of Finland.