Skip to content

Latest commit

 

History

History
78 lines (59 loc) · 4.46 KB

README.md

File metadata and controls

78 lines (59 loc) · 4.46 KB
Raw

Lifelong-MonoDepth: Lifelong Learning for Multi-Domain Monocular Metric Depth Estimation

We present an efficient multi-head framework that enables lifelong, cross-domain, and scare-aware monocular depth learning. Depth maps in the real world are significantly different across domains; their quality and scales are domain dependent. Therefore, the model has to assemble multiple prediction branches for multi-domain metric depth inference. To this end, we present a framework that consists of a domain-shared encoder and multiple domain-specific predictors. The framework allows robust metric depth learning across multi-domains. For more detailed information, please check our paper.

photo not available

Results

We show through experiments that the proposed method can

  • enable lifelong learning for scale-aware depth estimation,
  • cope with significant domain shift,
  • infer a depth map in real-time.

Qualitative comparisons:

photo not available

Datasets

We provided data index of RGB and depth pairs for both training and test set of all three datasets in csv file, you can find them in ./datasets/. Then, download the data required for training and test.

Trained Models

We provide all trained models as follows. We name the model in the order of learning.

Models Description
N.pth.tar Trained model on the NYU-v2
K.pth.tar Trained model on the KITTI
S.pth.tar Trained model on the ScanNet
NK.pth.tar Trained model on the NYU-v2 and KITTI
KN.pth.tar Trained model on the KITTI and NYU-v2
NKS.pth.tar Trained model on the NYU-v2, KITTI, and ScanNet
NSK.pth.tar Trained model on the NYU-v2, ScanNet, and KITTI
SNK.pth.tar Trained model on the ScanNet, NYU-v2, and KITTI
SKN.pth.tar Trained model on the ScanNet, KITTI, and NYU-v2
KSN.pth.tar Trained model on the KITTI, ScanNet, and NYU-v2
KNS.pth.tar Trained model on the KITTI, NYU-v2, and ScanNet

Running

  • Test

    Download a trained model that you want to test and put it into ./runs/. The default model is NKS.pth.tar.

    1. Testing with domain prior that assumes learning orders are given as a prior: python test.py
    2. Testing without domain prior that requires automatically select the domain-specific predictor during inference: python test_inference.py

  • Train

    The following shows examples of learning in the order of NYU-v2 → KITTI → ScanNet

    1. Training on a single-domain: python train_N.py
    2. Training on two domains: python train_NK.py
    3. Training on three domains: python train_NKS.py

Citation

@ARTICLE{10293000,
  author={Hu, Junjie and Fan, Chenyou and Zhou, Liguang and Gao, Qing and Liu, Honghai and Lam, Tin Lun},
  journal={IEEE Transactions on Neural Networks and Learning Systems}, 
  title={Lifelong-MonoDepth: Lifelong Learning for Multidomain Monocular Metric Depth Estimation}, 
  year={2023},
  volume={},
  number={},
  pages={1-11},
  doi={10.1109/TNNLS.2023.3323487}}