Bring back TorchBench AO dashboard #6101
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After pytorch/benchmark#2561, TorchBench AO benchmark data is now available to query and we can finally use that dashboard again. If this proves useful, the next steps would be:
autoquant
works, but notint8dynamic
andint8weightonly
. I'm not sure if they are still relevant, but if they are, ao team should know how to fix them (cc @jerryzh168). The run on TorchBench is at https://github.com/pytorch/benchmark/actions/workflows/torchao.ymlTesting
The two metrics speedup and abs execution time are now showing up https://torchci-git-fork-huydhn-ch-migrate-torchao-queries-fbopensource.vercel.app/benchmark/torchao?dashboard=torchao&startTime=Sun%2C%2015%20Dec%202024%2011%3A06%3A45%20GMT&stopTime=Sun%2C%2022%20Dec%202024%2011%3A06%3A45%20GMT&granularity=hour&mode=inference&dtype=autoquant&deviceName=cuda%20(a100)&lBranch=main&lCommit=07e6ef43fca2e95bc6cf59f97ba6251e618ef0e3&rBranch=main&rCommit=c03fa7c6c1bd03242a9de1fddb77a9c778106afd