epic: Replace optimizers or code new optimizers?

[tikikun]

Goal

There are a few new optimizers which can converge faster? Should consider switch?

Resources

https://arxiv.org/abs/2411.02853

[tikikun]

@tuanlda78202 you can also look into this

[tikikun]

Precision scaling

https://arxiv.org/abs/2411.04330

[tikikun]

Some observations:

• Higher precision can lead to better convergence
• The precision scaling paper made use of the same optimizer, we might get better result with better optimizer

[tikikun]

Current results:

• 1B llama can stabilize at higher learning rate on FP32 compared to bf16
• Catastrophic forgetting occred