EvInsMOS: Instance-Level Moving Object Segmentation from a Single Image with Events

This project is under construction, please wait for updates.

Requirements

The code has been tested with Python 3.9.13 and PyTorch 1.13.0 with CUDA 11.7.

conda create --name emos python=3.9
conda activate emos
pip install torch==1.13.0 torchvision==0.14.0 --index-url https://download.pytorch.org/whl/cu117
pip install tensorboard==2.11.0 pypng h5py omegaconf opencv-python func_timeout imageio einops pytorch_metric_learning
pip install pycocotools numpy==1.22 # pycocotools may need to downgrade numpy
pip install openmim
mim install mmcv mmdet

Datasets

If you just want to run this project, just download our pre-processed files here for EVIMO and EKubric datasets.

Download and uncompress them as follows

dataset/evimo
├── emos_preprocess/
├── val_movsegs_cocogt.json
dataset/ekubric
├── emos_preprocess_eb5p1/
├── moving_segmentation/
├── val_movsegs_cocogt.json
├── train_seqs.txt
├── val_seqs.txt

Optional. If it doesn't meet your needs,

you should first download the raw EVIMO dataset (npz and txt version) and EKubric dataset. Then process follows

python scripts/evimo_preprocess.py
python scripts/ekubric_preprocess.py

Pretrained Weights

Pretrained weights can be downloaded from Google Drive.

Please put them into the weights folder.

Evaluation

For EVIMO dataset

CUDA_VISIBLE_DEVICES="0" python eval.py -c ./conf/evimo.yaml --weights ./weights/EvInsMOS_EVIMO.pt -bs 4

Results

Statistics: mask_mae_0:0.017981, mask_iou_0:0.760991, mos_iou_0:0.720810, coco_mAP:0.368000, coco_mAP_50:0.721000, coco_mAP_75:0.352000, coco_mAP_s:0.145000, coco_mAP_m:0.457000, coco_mAP_l:0.501000, total_time:552.1754s, each_time:0.0461s, iters:1961, data_length:7844.

For EKubric dataset

CUDA_VISIBLE_DEVICES="0" python eval.py -c ./conf/ekubric.yaml --weights ./weights/EvInsMOS_EKubric.pt -bs 2

Results

Statistics: mask_mae_0:0.039794, mask_iou_0:0.833265, mos_iou_0:0.795460, epe:2.635708, epe_1px:53.184345, coco_mAP:0.485000, coco_mAP_50:0.601000, coco_mAP_75:0.501000, coco_mAP_s:0.007000, coco_mAP_m:0.213000, coco_mAP_l:0.629000, total_time:425.4213s, each_time:0.1480s, iters:1032, data_length:2063.

Training

For EVIMO dataset

Requires four 24G GPUs (4 RTX3090 we use) about 3 days.

CUDA_VISIBLE_DEVICES="0,1,2,3" python train.py -c ./conf/evimo.yaml

For EKubric dataset

Requires eight 24G GPUs (8 RTX3090 we use) about 3 days.

CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" python train.py -c ./conf/ekubric.yaml 

Acknowledgments

Thanks for the following helpful open source projects: mmdetection, Restormer, pytorch_metric_learning.