Skip to content

Commit

Permalink
[BE] [Perf optimization] Speed up queued_jobs query (#5945)
Browse files Browse the repository at this point in the history
Optimizes the query to use just 10% of the resources compared to before
by :
- Adding a max time range to limit results to just the relevant window
- Move the time range filter to the earlier query. Not sure why it makes
such a big difference, but it does. Maybe due to the lack of FINAL
there? Adding in FINAL tanks the performance

Perf change:
- Memory used: 9.7 GB -> 0.8 GB  
- Elapsed time: 3.5 s -> 0.4s
- Rows read: 12 million -> 8 million
  • Loading branch information
ZainRizvi authored Nov 20, 2024
1 parent a68eef5 commit e0d02f1
Showing 1 changed file with 5 additions and 4 deletions.
9 changes: 5 additions & 4 deletions torchci/clickhouse_queries/queued_jobs/query.sql
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
--- This query is used by HUD metrics page to get the list of queued jobs
with possible_queued_jobs as (
select id, run_id from default.workflow_job where status = 'queued'
select id, run_id
from default.workflow_job -- FINAL not needed since we just use this to filter a table that has already been FINALed
where status = 'queued'
AND created_at < (CURRENT_TIMESTAMP() - INTERVAL 5 MINUTE)
AND created_at > (CURRENT_TIMESTAMP() - INTERVAL 1 WEEK)
)
SELECT
DATE_DIFF(
Expand All @@ -27,9 +31,6 @@ WHERE
and workflow.id in (select run_id from possible_queued_jobs)
and workflow.repository.'full_name' = 'pytorch/pytorch'
AND job.status = 'queued'
AND job.created_at < (
CURRENT_TIMESTAMP() - INTERVAL 5 MINUTE
)
/* These two conditions are workarounds for GitHub's broken API. Sometimes */
/* jobs get stuck in a permanently "queued" state but definitely ran. We can */
/* detect this by looking at whether any steps executed (if there were, */
Expand Down

0 comments on commit e0d02f1

Please sign in to comment.