The Positional Encoding is not using sin / cos?

[mw66]

nanoGPT/model.py

Line 178 in 9755682

pos_emb = self.transformer.wpe(pos) # position embeddings of shape (t, n_embd)

E.g compared with the section Positional Encoding of the following article:

http://jalammar.github.io/illustrated-transformer/

So, did I miss something?

Is this an overlook, or simplification? And how does this affect the training result?

Anyone can help explain?

Thanks.

[jhauret]

From GPT-1 paper:

We used learned position embeddings instead of the sinusoidal version proposed in the original work.