Monkeypatch for Llama 3.2-Vision

[tyler-romero]

Summary

Add monkeypatch to support Llama 3.2-Vision models.

Details

Llama 3.2-Vision is a multimodal model. It is also only available in transformers>=4.45.0.

Torchvision is required to run the multimodal tests for Llama 3.2-Vision (the image processor requires it).

Testing Done

• Hardware Type: RTX 4090
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[tyler-romero]

Wanted to add support for LayerNorm in the vision tower as well, but it is hard to patch the way it is used in modeling_mllama.py - a patch would end up changing torch.nn globally.

[tyler-romero]

Just need someone to set a HF Token that has accepted the llama license in the CICD env:
https://discord.com/channels/1189498204333543425/1275130785933951039/1290378722653896744

>           raise EnvironmentError(
                "You are trying to access a gated repo.\nMake sure to have access to it at "
                f"[https://huggingface.co/{path_or_repo_id}.\n{str(e)}](https://huggingface.co/%7Bpath_or_repo_id%7D./n%7Bstr(e)%7D)"
            ) from e
E           OSError: You are trying to access a gated repo.
E           Make sure to have access to it at https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct.
E           401 Client Error. (Request ID: Root=1-66fc5343-253e99011960556f10a26970;1bfe5fb8-1fa8-4ff0-a21e-28041d5b6752)
E           
E           Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct/resolve/main/config.json.
E           Access to model meta-llama/Llama-3.2-11B-Vision-Instruct is restricted. You must have access to it and be authenticated to access it. Please log in.

[shivam15s]

Added HF hub read secret to the repo and got the gated llama 3.2 access

[tyler-romero]

Thanks, for some reason I'm still seeing the same issue - could you try triggering a cicd run on this branch?

[shivam15s]

Let me get back. Still figuring out how to configure the gpu to use the access token.

[shivam15s]

Seems like using gated stuff for tests isn't a good idea. I'm looking into using fake/mirrored models/processors for the tests

[tyler-romero]

Any update @shivam15s ?

[shivam15s]

Hey @tyler-romero, created a PR on your branch. Let me know if the proposed solution makes sense

[tyler-romero]

Really nice solution, thank you!

[shivam15s]

Seems like a convergence test fails because of a precision issue.
Could you please look into it? Thanks

[tyler-romero]

Now that you made custom tokenizers for the multimodal tests, I could reduce the vocab sizes of the mini-models (to be in-line with the vocab sizes of the other mini-models), and the convergence tests are passing now.

[shivam15s]

Great work, lgtm

[tyler-romero]

Sounds good, feel free to merge (only collaborators/linkedin employees can merge) - and please add yourself as a contributor to this change if GitHub doesn't do that automatically.

[ByronHsu]

Thank for both for driving this!!