This repository is the official implementation of the papers Continual SLAM and CoVIO:
Continual SLAM: Beyond Lifelong Simultaneous Localization and Mapping through Continual Learning
Niclas Vödisch, Daniele Cattaneo, Wolfram Burgard, and Abhinav Valada.
International Symposium on Robotics Research (ISRR), 2022
CoVIO: Online Continual Learning for Visual-Inertial Odometry
Niclas Vödisch, Daniele Cattaneo, Wolfram Burgard, and Abhinav Valada.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2023
If you find our work useful, please consider citing our papers:
@InProceedings{voedisch2023clslam,
author="V{\"o}disch, Niclas and Cattaneo, Daniele and Burgard, Wolfram and Valada, Abhinav",
editor="Billard, Aude and Asfour, Tamim and Khatib, Oussama",
title="Continual SLAM: Beyond Lifelong Simultaneous Localization and Mapping Through Continual Learning",
booktitle="Robotics Research",
year="2023",
publisher="Springer Nature Switzerland",
address="Cham",
pages="19--35",
}
@article{voedisch2023covio,
title="CoVIO: Online Continual Learning for Visual-Inertial Odometry",
author="V{\"o}disch, Niclas and Cattaneo, Daniele and Burgard, Wolfram and Valada, Abhinav",
journal="IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops",
year="2023"
}
CL-SLAM and CoVIO are also featured in the OpenDR toolkit.
While lifelong SLAM addresses the capability of a robot to adapt to changes within a single environment over time, in this paper we introduce the task of continual SLAM. Here, a robot is deployed sequentially in a variety of different environments and has to transfer its knowledge of previously experienced environments to thus far unseen environments, while avoiding catastrophic forgetting. This is particularly relevant in the context of vision-based approaches, where the relevant features vary widely between different environments. We propose a novel approach for solving the continual SLAM problem by introducing CL-SLAM. Our approach consists of a dual-network architecture that handles both short-term adaptation and long-term memory retention by incorporating a replay buffer. Extensive evaluations of CL-SLAM in three different environments demonstrate that it outperforms several baselines inspired by existing continual learning-based visual odometry methods.
Visual odometry is a fundamental task for many applications on mobile devices and robotic platforms. Since such applications are oftentimes not limited to predefined target domains and learning-based vision systems are known to generalize poorly to unseen environments, methods for continual adaptation during inference time are of significant interest. In this work, we introduce CoVIO for online continual learning of visual-inertial odometry. CoVIO effectively adapts to new domains while mitigating catastrophic forgetting by exploiting experience replay. In particular, we propose a novel sampling strategy to maximize image diversity in a fixed-size replay buffer that targets the limited storage capacity of embedded devices. We further provide an asynchronous version that decouples the odometry estimation from the network weight update step enabling continuous inference in real time. We extensively evaluate CoVIO on various real-world datasets demonstrating that it successfully adapts to new domains while outperforming previous methods.
Clone repository: git clone --recurse-submodules https://github.com/robot-learning-freiburg/CL-SLAM.git
- Create conda environment:
conda create --name continual_slam python=3.8 - Activate conda environment:
conda activate continual_slam - Install dependencies:
pip install -r requirements.txt - For smooth development, install git hook scripts:
pre-commit install
🔄 Install g2opy
We use g2o for pose graph optimization.
- Apply fixes for Eigen version >= 3.3.5:
./third_party/fix_g2opy.py - Install C++ requirements:
conda install cmakeconda install -c conda-forge eigenconda install -c conda-forge suitesparse
- Install g2opy:
cd third_party/g2opy
mkdir build
cd build
cmake -DPYBIND11_PYTHON_VERSION=3.8 ..
make -j8 |NOTE: reduce number if running out of memory
cd ..
|NOTE: remove any .so file which is not for Python 3.8
python setup.py install |NOTE: Ensure that the conda environment is active
To re-train or run the experiments from our paper, please download and pre-process the respective datasets.
Download the following files from https://www.cityscapes-dataset.com/downloads/:
leftImg8bit_sequence_trainvaltest.zip(324GB)timestamp_sequence.zip(40MB)vehicle_sequence.zip(56MB)disparity_sequence_trainvaltest.zip(106GB) (optionally, used for computing the depth error)
Download the RTK ground truth from https://robotcar-dataset.robots.ox.ac.uk/ground_truth/ (91MB).
Download the camera models from https://robotcar-dataset.robots.ox.ac.uk/downloads/ (129MB).
We used the files from https://robotcar-dataset.robots.ox.ac.uk/datasets/2015-08-12-15-04-18/:
2015-08-12-15-04-18_stereo_centre_01.tar,...0*.tar(25GB)2015-08-12-15-04-18_gps.tar(20MB)
Undistort the center images:
python datasets/robotcar.py <IMG_PATH> <MODELS_PATH>Download the KITTI Odometry data from http://www.cvlibs.net/datasets/kitti/eval_odometry.php:
odometry data set (color, 65 GB)odometry ground truth poses (4 MB)
Download the KITTI raw data from http://www.cvlibs.net/datasets/kitti/raw_data.php for the runs specified in datasets/kitti.py (search for KITTI_RAW_SEQ_MAPPING).
[synced+rectified data]
Download the ground truth depth from http://www.cvlibs.net/datasets/kitti/eval_depth_all.php (optionally, used for computing the depth error).
annotated depth maps data set (14GB)
Extract the raw data matching the odometry dataset. Note that sequence 03 is excluded as no IMU data (KITTI raw) has been released.
python datasets/kitti.py <RAW_PATH> <ODOMETRY_PATH> --oxts
python datasets/kitti.py <GT_DEPTH_PATH> <ODOMETRY_PATH> --depthWe pre-trained CL-SLAM on the Cityscapes Dataset.
You can either download the resulting weights, where we masked potentially dynamic objects, or pre-train the DepthNet and PoseNet by yourself by running our code.
Note that you have to adjust the dataset_path in config/config_pretrain.yaml.
python main_pretrain.pyModel weights: https://drive.google.com/file/d/1M3I_C3f4JWGLlLaeiyT8-QqoM4GctCw3/view?usp=sharing (Please unzip the file after download.)
For adaptation, we used the KITTI Odometry Dataset and the Oxford RobotCar Dataset.
The experiments in the paper are conducted on the KITTI sequences 09 and 10 as well as on two RobotCar sequences.
In order to fill the replay buffer with the pre-training data, please run the following script after having adjusted the paths in the file.
This can take some time.
python make_cityscapes_buffer.pyIn the configuration file config/config_adapt.yaml, please adjust the following parameters:
Dataset.dataset--> Set eitherKittiorRobotCarDataset.dataset_path--> Set the path to the dataDepthPosePrediction.load_weights_folder--> Should be the path to the weights from pre-training or the previous adaptationSlam.dataset_sequence--> Set the KITTI sequence, or1or2for RobotCarSlam.logging--> If this is set to true, make sure to enable dataloaders in theslam/slam.pyhavewith_depthsargument set toTrue, also make sure that you havegt_depthin your dataset folder
Then run:
python main_adapt.pyThe originally released code for Continual SLAM, i.e., without the extensions of CoVIO, can be found under commit 4ac27f6.
The asynchronous variant is provided in the OpenDR toolkit.
For academic usage, the code is released under the GPLv3 license. For any commercial purpose, please contact the authors.
This work was funded by the European Union’s Horizon 2020 research and innovation program under grant agreement No 871449-OpenDR.