Original Paper's Code Released

[xanderdunn]

The author of the original paper that the transformer MVP model was based on has release his code: https://github.com/gzerveas/mvts_transformer

It may be useful for testing differences in results compared to what the paper is reporting on the benchmark datasets.

[oguiza]

Hi @xanderdunn,
Thanks a lot for sharing!! I didn't know they'd released it.
I'm really interested in their comparing both TST and MVP to their codebase. I'm not sure I'll be able to do it this week though. If I can't, I'll do it in early September. I'll keep you posted.
I've you run any type of comparison, it'd be great if you shared your insights.
Thanks again!!

[oguiza]

I've just closed issue #170 with this comment:

I've just compared TST and TST implementations in tsai with the original code published by the authors (https://github.com/gzerveas/mvts_transformer). I've updated the models based on a few findings that were not described in the arxiv paper:

learnable positional encoding is initialized using: nn.init.uniform_(W_pos, -0.02, 0.02)
an additional dropout layer is inserted in the position-wise feed-forward sublayer of the encoder layers.
the final output is multiplied by the padding mask (to ensure padded positions are not taken into account)
the output of the encoder is passed through an activation layer ('gelu').
One key difference I've noticed is that in the original code masks are created in numpy, one at a time. This is certainly much slower than the mask creation processing MVP. MVP creates all masks for an entire batch on the GPU, which is much faster.

The code is already update in github.

Thanks @xanderdunn for informing us that the original code was already available!

Based on this, I'll close this issue.