OpenDistill3D: Open-World 3D Instance Segmentation with Unified Self-Distillation for Continual Learning and Unknown Class Discovery

(ECCV 2024)

Mohamed El Amine Boudjoghra¹, Jean Lahoud¹, Hisham Cholakkal¹, Rao Muhammad Anwer^1,2, Salman Khan^1,3, Fahad Khan^1,4

¹Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) ²Aalto University ³Australian National University ⁴Linköping University

News

• 05 July 2024: Code and trained models released. 💻
• 01 July 2024: Paper accepted into ECCV2024. 🥳

Abstract

Open-world 3D instance segmentation is a recently introduced problem with diverse applications, notably in continually learning embodied agents. This task involves segmenting unknown instances and learning new instances when their labels are introduced. However, prior research in the open-world domain has traditionally addressed the two sub-problems, namely continual learning and unknown object identification, separately. This approach has resulted in limited performance on unknown instances and cannot effectively mitigate catastrophic for- getting. Additionally, these methods bypass the utilization of the information stored in the previous version of the continual learning model, instead relying on a dedicated memory to store historical data samples, which inevitably leads to an expansion of the memory budget. In this paper, we argue that continual learning and unknown class identification should be tackled in conjunction. Therefore, we propose a new exemplar- free approach for 3D continual learning and the discovery of unknown classes through self-distillation. Our approach leverages the pseudo-labels generated by the model from the preceding task to improve the unknown predictions during training while simultaneously mitigating catastrophic forgetting. By integrating these pseudo-labels into the continual learning process, we achieve enhanced performance in handling unknown classes. We validate the efficacy of the proposed approach via comprehensive experiments on various splits of the ScanNet200 dataset, showcasing superior performance in continual learning and unknown class retrieval compared to the state-of-the-art.

Figure I: Proposed open-world 3D instance segmentation pipeline.

Installation guide

Kindly check Installation guide on how to setup the Conda environment and how to preprocess ScanNet200 dataset.

Training

<SPLIT_ID> takes the following values A, B, or C.

sh scripts/train/<SPLIT_ID>/opendistill3d.sh

The evaluation results throughout the training will be stored in ./saved/<EXPERIMENT_NAME>/<TASK_ID>/ow_results/<EPOCH>_ow_results.yml

Download trained models and data

• Use this link for the checkpoints to reproduce our results in Table 1 in the paper.
• Use this link for the checkpoints to reproduce our results in Table 7 in the paper.
• Use this link to download the ScanNet200 sampled points used for training models in Table 7 in the paper.

Evaluation

<TASK_ID> takes the following values task1, task2, or task3

sh evaluate.sh <SPLIT_ID> <TASK_ID> <CHECKPOINT_PATH>

BibTeX 🙏