Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Z Loss in CE #197

Open
Fr0do opened this issue Sep 3, 2024 · 2 comments · May be fixed by #239
Open

Z Loss in CE #197

Fr0do opened this issue Sep 3, 2024 · 2 comments · May be fixed by #239
Assignees
@shivam15s
Labels
feature

Comments

@Fr0do
Copy link

Fr0do commented Sep 3, 2024

🚀 The feature, motivation and pitch

Often used in pretraining of LMs for stabilization, i.e. the recent Chameleon & PaLM.

Alternatives

flash-attn has implementations of abovementioned features, however, does not support fusing with linear head.

Additional context

No response
@ByronHsu
Copy link
Collaborator

ByronHsu commented Sep 3, 2024

Legit ask! We have tracked smooth label at #81. I modify the title for only Z loss to prevent duplication.

@ByronHsu ByronHsu changed the title Z Loss & Smooth Labels in CE Z Loss in CE Sep 3, 2024
@Tcc0403
Copy link
Contributor

Tcc0403 commented Sep 6, 2024

@ByronHsu #take
To support z loss, I just need a little add-ons to #198.
I'll work on it after merging label_smoothing PR.

This was referenced Sep 9, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shivam15s shivam15s

Labels
feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support Z Loss in CE Tcc0403/Liger-Kernel
Support Z Loss in CE Tcc0403/Liger-Kernel
5 participants
@Fr0do @ByronHsu @qingquansong @shivam15s @Tcc0403