Improve master_commit_red query performance

[clee2000]

Pre filter the commits so we can filter the workflow job and workflow run tables on it later

This improves speed for all time ranges up to 1 year. I did not check beyond that
I believe the memory used is about the same, but it scans more rows for some reason

This is the query behind this chart

on the metrics page

Tentative wins, sample size 3

+------+----------+-----------+-------------+---------------+--------------+---------------+----------------+--------------+
| Test | Avg Time | Base Time | Time Change | % Time Change |   Avg Mem    |    Base Mem   |   Mem Change   | % Mem Change |
+------+----------+-----------+-------------+---------------+--------------+---------------+----------------+--------------+
|  0   |   1163   |   23117   |    -21954   |      -95      | 157241294.6  | 1539051189.75 | -1381809895.15 |     -90      |
|  1   |   1281   |   14509   |    -13228   |      -91      |  350569557   |  1585082143.8 | -1234512586.8  |     -78      |
|  2   |   8704   |   22954   |    -14250   |      -62      | 1774302065.6 |   3929546293  | -2155244227.4  |     -55      |
+------+----------+-----------+-------------+---------------+--------------+---------------+----------------+--------------+

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
torchci	✅ Ready (Inspect)	Visit Preview	Jan 22, 2025 7:55pm

[huydhn]

This reads a bit confusing IMO. If I read it correctly, the all_runs table has the alias as workflow_run, which has the correct syntax. But I always think of workflow_run as the workflow_run table instead of having it as an alias. So, it feels easier just to call it all_runs

[huydhn]

We can only see the improvement on https://hud.pytorch.org/query_execution_metrics after this lands. I wonder if there is a way to get the information for the new query at PR time. Let's chat more on this when you're back

[ZainRizvi]

Nice discovery that having separating table filters before doing the join yields perf improvements.

To show how much of a difference it really made could you update the PR description with the stats? (e.g. old/new duration, memory usage)

[ZainRizvi]

do pushes ever get updated? Wondering if we really need the perf hit from FINAL

[ZainRizvi]

Another optimization opportunity:

Do we really need to know when a job is pending? If not, instead of FINAL we can filter on a terminal conclusion

[clee2000]

I'm going to leave it as is because I don't want to change the results from the old query