Continual Learning in Recurrent Neural Networks

A continual learning approach for recurrent neural networks that has the flexibility to learn a dedicated set of parameters, fine-tuned for every task, that doesn't require an increase in the number of trainable weights and is robust against catastrophic forgetting.

For details on this approach please refer to our paper. In addition, a short overview is given in this talk and we summarize some key points from our paper in this talk. Experiments on continual learning with hypernetworks using static data can be found in this repository.

If you are interested in working with hypernetworks in PyTorch, check out the package hypnettorch.

Copy Task Experiments

You can find instructions on how to reproduce our experiments on all Copy Task variants and on how to use the corresponding code in the subfolder sequential.copy.

Part-of-Speech Tagging Experiments

You can find instructions on how to reproduce our PoS-Tagging experiments and on how to use the corresponding code in the subfolder sequential/pos_tagging.

Stroke MNIST Experiments

You can find instructions on how to reproduce our Stroke MNIST experiments and on how to use the corresponding code in the subfolder sequential.smnist.

Sequential Stroke MNIST Experiments

You can find instructions on how to reproduce our Sequential Stroke MNIST experiments and on how to use the corresponding code in the subfolder sequential.seq_smnist.

Audioset Experiments

You can find instructions on how to reproduce our Audioset experiments and on how to use the corresponding code in the subfolder sequential.audioset.

Documentation

Please refer to the README in the subfolder docs for instructions on how to compile and open the documentation.

Setup Python Environment

We use conda to manage Python environments. To create an environment that already fulfills all package requirements of this repository, simply execute

$ conda env create -f environment.yml
$ conda activate cl_rnn_env

Citation

Please cite our paper if you use this code in your research project.

@inproceedings{ehret2020recurrenthypercl,
  title={Continual Learning in Recurrent Neural Networks},
  author={Benjamin Ehret and Christian Henning and Maria R. Cervera and Alexander Meulemans and Johannes von Oswald and Benjamin F. Grewe},
  booktitle={International Conference on Learning Representations},
  year={2021},
  url={https://arxiv.org/abs/2006.12109}
}