[2023-08-19] ROFusion is released.
This work is based on our arXiv tech report, which is going to appear in ICANN 2023. We propose a hybrid point-wise Radar-Optical fusion approach for object detection in autonomous driving scenarios. The framework benefits from dense contextual information from both the range-doppler spectrum and images which are integrated to learn a multi-modal feature representation. Furthermore, we propose a novel local coordinate formulation, tackling the object detection task in an object-centric coordinate. Extensive results show that with the information gained from optical images, we could achieve leading performance in object detection (97.69% recall) compared to recent state-of-the-art methods FFT-RadNet (82.86% recall). Ablation studies verify the key design choices and practicability of our approach given machine generated imperfect detections.
Pre-trained models can be downloaded here
If you find our work useful in your research, please consider citing:
@article{liu2023rofusion,
title={ROFusion: Efficient Object Detection using Hybrid Point-wise Radar-Optical Fusion},
author={Liu, Liu and Zhi, Shuaifeng and Du, Zhenhua and Liu, Li and Zhang, Xinyu and Huo, Kai and Jiang, Weidong},
journal={arXiv preprint arXiv:2307.08233},
year={2023}
}
This repo uses the RADIal dataset. Download the RADIal dataset as explained here.
We pre-encode the dataset as follows:
cd PreEncoder
python predatase.py --config ../ROFusion_dataset.json --detector ${detector}The YOLO label can be download here.
YOLO detection results are provided by YOLOX
There will generate 4 new dataset folders:
- Pre_PCL: Nontrivial radar point cloud that in image 2D bounding boxes.
- Center_Label: Encoded object center labels (rb, ab, rm, am)
- Point_Label: Nontrivial encoded radar point cloud labels
- Box_Radar_PCL: Radar point cloud that in image 2D bounding boxes.
We also provide the data generated after pre-encoding:
All the codes are tested in the following environment:
- Linux (tested on Ubuntu 20.04)
- Python 3.9
- Pytorch 1.11.0, Torchvision 0.12.0, CUDA 11.3, cuDNN 8.2
- Clone the repository:
git@github.com:LiuLiu-55/ROFusion.git - Create a new conda environment:
conda create -n ROFusion python==3.9.7 - Activate the environment using:
conda activate ROFusion - Install the pytorch:
conda install pytorch==1.11.0 torchvision==0.12.0 cudatoolkit==11.3.1 cudnn=8.2.0 -c pytorch - Install the requirements:
pip install -r requirements.txt
python Train.py --config config/ROFusion.json- Test with a pretrained model
python Test.py --config config/ROFusion.json --checkpoint ROFusion_ResNet18_epoch24_loss_13.9696_AP_0.9914_AR_0.9914.pth- Evaluation with a pretrained model
python Evaluation.py --config config/ROFusion.json --checkpoint ROFusion_ResNet18_epoch24_loss_13.9696_AP_0.9914_AR_0.9914.pth --detector ${Detector}--detector is the option whether you use the YOLO 2D detection results.
ROFusion is released under the Apache 2.0 license. See LICENSE.txt for more information.