Handle benchmark configs when extracting benchmark results

[huydhn]

The benchmark configs JSON are uploaded as artifacts and can be used later to populate the benchmark results with the model name and configs before uploading to the database.

• I'll submit another PR to clean up the script later because the v2 schema is not used anymore after this.
• We also need a corresponding PR on the dashboard side Add support for benchmark configs on ExecuTorch dashboard test-infra#6110

Testing

https://torchci-git-fork-huydhn-add-executorch-backend-fbopensource.vercel.app/benchmark/llms?startTime=Tue%2C%2017%20Dec%202024%2019%3A05%3A31%20GMT&stopTime=Tue%2C%2024%20Dec%202024%2019%3A05%3A31%20GMT&granularity=hour&lBranch=handle-benchmark-config-dashboard&lCommit=c48da2bd2c9a32705db9b1adf638344474c275a4&rBranch=handle-benchmark-config-dashboard&rCommit=c33f815eff17c8890f2c8527dc0f0dbca50b4397&repoName=pytorch%2Fexecutorch&modelName=All%20Models&backendName=All%20Backends&dtypeName=All%20DType&deviceName=All%20Devices

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7433

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Unrelated Failures

As of commit c33f815 with merge base 82763a9 ():

NEW FAILURE - The following job has failed:

• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 24f8d51aa40bfbda647dcfec6d80b48b5e109c1597eb28c826663bd3a37f2905 /exec failed with exit code 92

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / test-demo-ios / mobile-job (ios) (gh) (trunk failure)
  Final attempt failed. Child_process exited with error code 1
• apple-perf / benchmark-on-device (llama, mps, apple_iphone_15, arn:aws:devicefarm:us-west-2:308535385114:devic... / mobile-job (ios) (gh) (trunk failure)
  Final attempt failed. Child_process exited with error code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@huydhn has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[huydhn]

Once this lands, we can wait for a week for old data to go out of focus on http://localhost:3000/benchmark/llms?repoName=pytorch%2Fexecutorch.

[guangy10]

@huydhn It looks like test-spec specialization starts failing last Fri since this PR merged. For example: https://github.com/pytorch/executorch/actions/runs/12540463242/job/34967784456.

[huydhn]

Oh, silly me. This failed for meta-llama/Llama-3.2-1B_llama3_fb16 because I tried to use that as the filename. It has an illegal '/' character, let me push a fix for this.