This project provides a set of examples with popular continual learning strategies and baselines. You can easily run experiments to reproduce results from original paper or tweak the hyperparameters to get your own results. Sky is the limit!
To guarantee fair implementations, we rely on the Avalanche library, developed and maintained by ContinualAI. Feel free to check it out and support the project!
The tables below describes all the experiments currently implemented in the experiments
folder, along with their result.
The tables are not meant to compare different methods but rather as a reference for their performance. Different methods may use
slightly different setups (e.g., starting from a pre-trained model or from scratch), so it does not always make sense to compare them.
If an experiment reproduces exactly the results of a paper in terms of Performance
(even if with different hyper-parameters), it is marked with ✅ on the Reproduced
column. Otherwise, it is marked with ❌.
Avalanche
means that we could not find any specific paper as reference and we used the performance of Avalanche obtained when the strategy was first add to the library.
If the Performance
is much worse than the expected one, the bug
tag is used in the Reproduced
column.
Finally, the Reference
column reports the expected performance, together with a link to the associated paper (if any). Note that the link does not always point to the paper which introduced the strategy, since it sometimes differs from the one we used to get the target performance.
ACC means the Average Accuracy on all experiences after training on the last experience.
First, we report the results for the non-online continual learning case (a.k.a. batch continual learning). Then, we report the results for the online continual learning case.
Benchmarks | Strategy | Scenario | Performance | Reference | Reproduced |
---|---|---|---|---|---|
Permuted MNIST | Less-Forgetful Learning (LFL) | Domain-Incremental | ACC=0.88 | ACC=0.88 | ✅ Avalanche |
Permuted MNIST | Elastic Weight Consolidation (EWC) | Domain-Incremental | ACC=0.83 | ACC=0.94 | ❌ |
Permuted MNIST | Synaptic Intelligence (SI) | Domain-Incremental | ACC=0.83 | ACC=0.95 | ❌ |
Split CIFAR-100 | LaMAML | Task-Incremental | ACC=0.70 | ACC=0.70 | ✅ |
Split CIFAR-100 | iCaRL | Class-Incremental | ACC=0.48 | ACC=0.50 | ✅ |
Split CIFAR-100 | Replay | Class-Incremental | ACC=0.32 | ACC=0.32 | ✅ Avalanche |
Split MNIST | RWalk | Task-Incremental | ACC=0.99 | ACC=0.99 | ✅ |
Split MNIST | Synaptic Intelligence (SI) | Task-Incremental | ACC=0.97 | ACC=0.97 | ✅ |
Split MNIST | GDumb | Class-Incremental | ACC=0.97 | ACC=0.97 | ✅ |
Split MNIST | GSS_greedy | Class-Incremental | ACC=0.82 | ACC=0.78 | ❌ |
Split MNIST | Generative Replay (GR) | Class-Incremental | ACC=0.75 | ACC=0.75 | ✅ |
Split MNIST | Learning without Forgetting (LwF) | Class-Incremental | ACC=0.23 | ACC=0.23 | ✅ |
Split Tiny ImageNet | LaMAML | Task-Incremental | ACC=0.54 | ACC=0.66 | ❌ |
Split Tiny ImageNet | Learning without Forgetting (LwF) | Task-Incremental | ACC=0.44 | ACC=0.44 | ✅ |
Split Tiny ImageNet | Memory Aware Synapses (MAS) | Task-Incremental | ACC=0.40 | ACC=0.40 | ✅ |
Split Tiny ImageNet | PackNet | Task-Incremental | ACC=0.46 | ACC=0.47 (Table 4 SMALL ) |
✅ |
Benchmarks | Strategy | Scenario | Performance | Reference | Reproduced |
---|---|---|---|---|---|
CORe50 | Deep Streaming LDA (DSLDA) | Class-Incremental | ACC=0.79 | ACC=0.79 | ✅ |
Permuted MNIST | GEM | Domain-Incremental | ACC=0.80 | ACC=0.83 | ✅ |
Split CIFAR-10 | Online Replay | Class-Incremental | ACC=0.50 | ACC=0.50 | ✅ Avalanche |
Split CIFAR-10 | ER-AML | Class-Incremental | ACC=0.47 | ACC=0.47 | ✅ |
Split CIFAR-10 | ER-ACE | Class-Incremental | ACC=0.45 | ACC=0.52 | ✅ |
Split CIFAR-10 | Supervised Contrastive Replay (SCR) | Class-Incremental | ACC=0.36 | ACC=0.48 | ✅ Avalanche |
Permuted MNIST | Average GEM (AGEM) | Domain-Incremental | ACC=0.81 | ACC=0.81 | ✅ |
Split CIFAR-100 | GEM | Task-Incremental | ACC=0.63 | ACC=0.63 | ✅ |
Split CIFAR-100 | Average GEM (AGEM) | Task-Incremental | ACC=0.62 | ACC=0.62 | ✅ |
Split CIFAR-100 | ER-ACE | Class-Incremental | ACC=0.24 | ACC=0.25 | ✅ |
Split CIFAR-100 | ER-AML | Class-Incremental | ACC=0.24 | ACC=0.24 | ✅ |
Split CIFAR-100 | Online Replay | Class-Incremental | ACC=0.21 | ACC=0.21 | ✅ Avalanche |
Split MNIST | CoPE | Class-Incremental | ACC=0.93 | ACC=0.93 | ✅ |
Split MNIST | Online Replay | Class-Incremental | ACC=0.92 | ACC=0.92 | ✅ Avalanche |
Outside Python standard library, the main packages required to run the experiments are PyTorch, Avalanche and Pandas.
- Avalanche: The latest version of this repo requires the latest Avalanche version (from master branch):
pip install git+https://github.com/ContinualAI/avalanche.git
. The CL baselines repo is tagged with the supported Avalanche version (you can browse the tags to check out all the versions). You can install the corresponding Avalanche versions withpip install avalanche-lib==[version number]
, where[version number]
is of the form0.1.0
. For some strategies (e.g., LaMAML) you may need to install Avalanche with extra packages, likepip install avalanche-lib[extra]
. For more details on how to install Avalanche, please check out the complete guide here. - PyTorch: we recommend to follow the official guide.
- Pandas:
pip install pandas
. Official guide.
Place yourself into the project root folder.
Experiments can be run with a python script by simply importing the function from the experiments
folder and executing it.
By default, experiments will run on GPU, when available.
The input argument to each experiment is an optional dictionary of parameters to be used in the experiments. If None
, default
parameters (taken from original paper) will be used.
from experiments.split_mnist import synaptic_intelligence_smnist # select the experiment
# can be None to use default parameters
custom_hyperparameters = {'si_lambda': 0.01, 'cuda': -1, 'seed': 3}
# run the experiment
result = synaptic_intelligence_smnist(custom_hyperparameters)
# dictionary of avalanche metrics
print(result)
Place yourself into the project root folder.
You should add the project root folder to your PYTHONPATH.
For example, on Linux you can set it up globally:
export PYTHONPATH=${PYTHONPATH}:/path/to/continual-learning-baselines
or just for the current command:
PYTHONPATH=${PYTHONPATH}:/path/to/continual-learning-baselines command to be executed
You can run experiments directly through console with the default parameters.
Open the console and run the python file you want by specifying its path.
For example, to run Synaptic Intelligence on Split MNIST:
python experiments/split_mnist/synaptic_intelligence.py
To execute experiment with custom parameters, please refer to the previous section.
Place yourself into the project root folder.
You can run all tests with
python -m unittest
or you can specify a test by providing the test name in the format tests.strategy_class_name.test_benchmarkname
.
For example to run Synaptic Intelligence on Split MNIST you can run:
python -m unittest tests.SynapticIntelligence.test_smnist
If you used this repo you automatically used Avalanche, please remember to cite our reference paper published at the CLVision @ CVPR2021 workshop: "Avalanche: an End-to-End Library for Continual Learning". This will help us make Avalanche better known in the machine learning community, ultimately making it a better tool for everyone:
@InProceedings{lomonaco2021avalanche,
title={Avalanche: an End-to-End Library for Continual Learning},
author={Vincenzo Lomonaco and Lorenzo Pellegrini and Andrea Cossu and Antonio Carta and Gabriele Graffieti and Tyler L. Hayes and Matthias De Lange and Marc Masana and Jary Pomponi and Gido van de Ven and Martin Mundt and Qi She and Keiland Cooper and Jeremy Forest and Eden Belouadah and Simone Calderara and German I. Parisi and Fabio Cuzzolin and Andreas Tolias and Simone Scardapane and Luca Antiga and Subutai Amhad and Adrian Popescu and Christopher Kanan and Joost van de Weijer and Tinne Tuytelaars and Davide Bacciu and Davide Maltoni},
booktitle={Proceedings of IEEE Conference on Computer Vision and Pattern Recognition},
series={2nd Continual Learning in Computer Vision Workshop},
year={2021}
}
We are always looking for new contributors willing to help us in the challenging mission of providing robust experiments to the community. Would you like to join us? The steps are easy!
- Take a look at the opened issues and find yours
- Fork this repo and write an experiment (see next section)
- Submit a PR and receive support from the maintainers
- Merge the PR, your contribution is now included in the project!
- Create the appropriate script into
experiments/benchmark_folder
. If the benchmark is not present, you can add one. - Fill the
experiment.py
file with your code, following the style of the other experiments. The script should return the metrics used by the related test. - Add to
tests/target_results.csv
the expected result for your experiment. You can add a number or a list of numbers. - Write the unit test in
tests/strategy_folder/experiment.py
. Follow the very simple structure of existing tests. - Update table in
README.md
.
- Place yourself into the avalanche folder and make sure you are using the avalanche version from that repository
in your python environment (it is usually enough to add
/path/to/avalanche
to yourPYTHONPATH
). - Use the
gitbisect_test.sh
(provided in this repository) in combination withgit bisect
to retrieve the avalanche commit introducing the regression.
git bisect start HEAD v0.1.0 -- # HEAD (current version) is bad, v0.1.0 is good
git bisect run /path/to/gitbisect_test.sh /path/to/continual-learning-baselines optional_test_name
git bisect reset
- The
gitbisect_test.sh
script requires a mandatory parameter pointing to thecontinual-learning-baselines
directory and an optional parameter specifying the path to a particular unittest (e.g.,tests.EWC.test_pmnist
). If the second parameter is not given, all the unit tests will be run. - The terminal output will tell you which commit introduced the bug
- You can change the
HEAD
andv0.1.0
ref to any avalanche commit.