-
Notifications
You must be signed in to change notification settings - Fork 181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add gemma lightning example for single L40 GPU #120
Conversation
9b1aa0b
to
dcd537a
Compare
@yundai424 @ByronHsu Waiting for #111 to check in and then can patch fused CE to test memory |
@qingquansong do you think if we should just replace llama with gemma2 in this lightning example, and fix the config so that it works with L40 GPU, just to minimize the code 🤔 -- and the example can be used out of the box as lightning studio example |
SG! Or maybe set the default one to be Gemma and provide a llama example in readme with config changes? (basically wanna keep the small changes (tokenizer and fsdp) to support llama in the pro lightning testing |
41793ce
to
a452e13
Compare
a452e13
to
d33b3d5
Compare
d0e7b41
to
b07fda0
Compare
Summary
Add gemma lightning example for single L40 GPU
Testing Done
make test
to ensure correctnessmake checkstyle
to ensure code stylemake test-convergence
to ensure convergence