The Dark Secrets of BERT | Text Machine Blog

[utterances-bot]

The Dark Secrets of BERT | Text Machine Blog

BERT and its Transformer-based cousins are still ahead on all NLP leaderboards. But how much do they actually understand about language?

https://text-machine-lab.github.io/blog/2020/bert-secrets/

[yuvalpinter]

Fascinating, useful stuff! I have a question about the last study - were attention heads ablated before fine-tuning, so just the LM properties were affected, or after fine-tuning (but before test)?

[ruanchaves]

I'm curious to know what the experimental results would be like on the SuperGLUE Benchmark, as it's supposed to be somewhat liberated from the biases and artifacts which are probably being exploited by randomly initialized BERT on the standard GLUE datasets. https://super.gluebenchmark.com/

[guacamolia]

@ruanchaves I agree, that would be indeed an interesting experiment to run, at the time of our submission we only had GLUE available.
@yuvalpinter thanks! the heads were ablated after fine-tuning.

[pramitchoudhary]

Great summarization here.

[charlesmartin14]

Are your pretrained and finetuned BERT models available for independent analysis ?

[guacamolia]

@charlesmartin14 we haven't released the models themselves, but we consistently used the scripts provided by huggingface to train all of the models.

[ghk829]

Wow the greatest article of the year!
I wonder what’s the black magic!!

[aswin-giridhar]

Thanks a lot for the paper summarization.