Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bring back TorchBench AO dashboard #6101

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented Dec 22, 2024

After pytorch/benchmark#2561, TorchBench AO benchmark data is now available to query and we can finally use that dashboard again. If this proves useful, the next steps would be:

  1. Resurrect ao benchmark on AWS A100 runner benchmark#2561 only brings back one example model for each suite (TorchBench, HF, TIMM). We need to add more.
  2. TorchBench AO dashboard shares the code with TorchInductor dashboard. While the former has been migrated to the new benchmark database, the latter hasn't. I will need to do that and clean this up in a later PR.
  3. Looking at the results on the dashboard, it seems that autoquant works, but not int8dynamic and int8weightonly. I'm not sure if they are still relevant, but if they are, ao team should know how to fix them (cc @jerryzh168). The run on TorchBench is at https://github.com/pytorch/benchmark/actions/workflows/torchao.yml

Testing

The two metrics speedup and abs execution time are now showing up https://torchci-git-fork-huydhn-ch-migrate-torchao-queries-fbopensource.vercel.app/benchmark/torchao?dashboard=torchao&startTime=Sun%2C%2015%20Dec%202024%2011%3A06%3A45%20GMT&stopTime=Sun%2C%2022%20Dec%202024%2011%3A06%3A45%20GMT&granularity=hour&mode=inference&dtype=autoquant&deviceName=cuda%20(a100)&lBranch=main&lCommit=07e6ef43fca2e95bc6cf59f97ba6251e618ef0e3&rBranch=main&rCommit=c03fa7c6c1bd03242a9de1fddb77a9c778106afd

@huydhn huydhn requested review from kit1980 and jerryzh168 December 22, 2024 11:02
Copy link

vercel bot commented Dec 22, 2024

@huydhn is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 22, 2024
Copy link

vercel bot commented Dec 22, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
torchci ✅ Ready (Inspect) Visit Preview Dec 22, 2024 11:26am

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants