Enable composable benchmark configs for flexible model+device+optimization scheduling

[guangy10]

It's not true that any combination of model + delegate can work, which make adding a new model to continuous run not easy as the workflow will run it across all delegates. Besides, each delegate may run with different configurations. For example, llama3.2 spinquant is using a prequantized checkpoitn hence it's not using the recipe for the regular fp32 checkpint. To support various combinations and optimizations, we are migrating to use benchmark_configs which is a set of predefined configs with combination of all optimizations that are possibly applied to the model, e.g. kv cache, embedding/activation quant, dtype, delegation, sdpa, etc.

In this PR, given a model (either a Hugging Face model ID or a in-tree model name) and a target platform ("android" vs "ios"), it's retrieving a list of supported benchmark configurations from the script gather_benchmark_configs.py and schedule the benchmark jobs accordingly. From the workflow dispatcher, users will just need to enter the model names, it will discover all supported benchmark configs for each model. Further more (not included in this PR), we can potentially expose config_args (key-value paris) from the script, if there is a way to store them in the DB and display in the dashboard. It will help understand how exactly a model is exported/lowered when discussing/debugging perf metrics.

Apple: https://github.com/pytorch/executorch/actions/runs/12404655922
Android: https://github.com/pytorch/executorch/actions/runs/12404651330

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7349

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit c9f0156 with merge base 6ab4399 ():

NEW FAILURE - The following job has failed:

• apple-perf / benchmark-on-device (llama, mps, apple_iphone_15, arn:aws:devicefarm:us-west-2:308535385114:devic... / mobile-job (ios) (gh)
  Final attempt failed. Child_process exited with error code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[huydhn]

We will probably not include this path in the continuous run until the RAM limits is lifted.

I'm still following up with AWS on this. The confusing part is I'm not entirely sure if this is something from AWS (but let's see). From what I read in https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.kernel.increased-memory-limit, this feels like a maybe and not a guarantee

If you use this entitlement, make sure your app still behaves correctly if additional memory isn’t available.

and

An increased memory limit is only available on some device models, call the os_proc_available_memory function to determine the amount of memory available

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[guangy10]

Core ML ANE job (ios 17): https://github.com/pytorch/executorch/actions/runs/12403482481

[guangy10]

Qualcomm HTP job: https://github.com/pytorch/executorch/actions/runs/12403453453/job/34626940914

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[guangy10]

@huydhn FYI, Core ML ANE and Qualcomm HTP jobs are not going to block this PR. I will merge this PR tonight to unblock you, and leave adding those paths separately if they are not working out of the box

[huydhn]

@huydhn FYI, Core ML ANE and Qualcomm HTP jobs are not going to block this PR. I will merge this PR tonight to unblock you, and leave adding those paths separately if they are not working out of the box

Sounds good! Once this lands, I could start working on bringing the benchmark config to the dashboard

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[guangy10]

Core ML ANE and Qualcomm HTP are not ready. Will enable in separate PRs. For now, we just hide these paths from benchmark_configs.

[guangy10]

@kirklandsign FYI, the results of newly added llama3 benchmarkings (spintquant, qlora, bf16) are also missing from the benchmark_results.json. See the upload job here: https://github.com/pytorch/executorch/actions/runs/12404651330/job/34631545501