Implement New State of the Art Self-Supervised Architectures

[xanderdunn]

tsai primarily implements BERT-like and InceptionTime architectures for self-supervised learning. BERT was designed for NLP tasks and InceptionTime is an ensemble of CNNs, which were designed for image tasks.

There are new state of the art models for self-supervised learning on images: SwAV and ReLIC. I believe SwAV is currently considered state of the art. ReLIC with stronger augmentations may be superior to SwAV. These architectures were trained and tested on image datasets, but, similar to the generalization of BERT and CNNs, they may generalize well to time series data.

This was intended more as an informational message to @oguiza than an issue to keep open until complete. Even if this is interesting to implement, it's a lot of work. I'll have to stick to using MVP and InceptionTimePlus for the time being, but I might get around to implementing one of these newer architectures after I've completed work using the existing architectures.

[oguiza]

Hi @xanderdunn,

(I've moved this to the discussion forum since it's not an open issue. )

Thanks for raising this interesting idea. Thanks for mentioning ReLIC. I hadn't heard it before.

Let me clarify something up front. MVP (TSBERT) works for any type of model that creates features of shape n_samples x features x seq_len. In tsai that includes all models with the 'Plus' suffix. MVP is actually inspired by the Time Series Transformer paper that uses a Transformer instead of a CNN. I adapted it to work with any type of model. So it's model agnostic.

I'm very interested in methods that leverage unlabeled time series. I've always wondered whether time series would benefit more from NLP or Vision semi-supervised/self-supervised methods. It'd be great to compare these methods.

Actually, I've used self-training and noisy student. Both of them have performed well (they usually improve baseline performance). I have also developed a prototype of Swav (still testing it) and plan to develop Barlow Twins for time series as well. It'd be great if more of us are interested in these methods and collaborate.

[xanderdunn]

Thanks for creating this discussion @oguiza, this is the perfect place for it. Please let me know how your SwAV experiments go. When you have code for any new models, I'll happily try them out on benchmarks as well as my larger than memory datasets to see how they perform. I'm eager to collaborate on any new self-supervised techniques applied to time series data.

[vrodriguezf]

Thank you @oguiza and @xanderdunn for working on this, it is definitely one of the hottest topics today in ml. Even if papers like TST are rejected in the academy because they are "just" translating concepts from vision and text to time series, in the real world I find this incredibly relevant.