Monkey patch layer norm in mllama #302
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ) | ||
|
|
||
| from liger_kernel.transformers.model.mllama import lce_forward as mllama_lce_forward | ||
|
|
||
| if rope: | ||
| modeling_mllama.apply_rotary_pos_emb = liger_rotary_pos_emb | ||
| if layer_norm: | ||
| modeling_mllama.nn.LayerNorm = LigerLayerNorm |
There was a problem hiding this comment.
FWIW I looked into this but was worried this would modify torch.nn.LayerNorm globally, not just for the targeted model
There was a problem hiding this comment.
That probably answers why ci tests fail. LigerLayerNorm gets applied even before we monkey patch because a previous test patched layer norm globally.
shivam15s
force-pushed
the
shisahni/patch-mllama
branch
from
945f810
to
e649305
Compare
shivam15s
changed the title
qingquansong
approved these changes
Oct 17, 2024
ByronHsu
approved these changes
Oct 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Monkey patches layer norm in mllama for conditional generation
Testing Done
Tested monkey patching works as intended
make testto ensure correctnessmake checkstyleto ensure code stylemake test-convergenceto ensure convergence