-
Notifications
You must be signed in to change notification settings - Fork 88
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Migrate LLM dashboards to v3 (#6057)
This migrates the queries from `oss_ci_benchmark_v2` to `oss_ci_benchmark_v3`. This will unblock the following updates: * Add delegation backend support for [ExecuTorch dashboard](https://hud.pytorch.org/benchmark/llms?repoName=pytorch%2Fexecutorch) instead of bundling it together with the model name `edsr coreml_all` (cc @guangy10) * Add LLM AO dashboard, whose data is only available in `oss_ci_benchmark_v3` (cc @jerryzh168) Some minor fixes that go with this: * Change the parameters to `benchmarks` and `models`. This is clearer than `filenames` and `names` * Remove `getJobId` param. Its origin is from the TorchInductor query when the job ID is a string. In `oss_ci_benchmark_v3`, the job is an UInt64, so there is no saving there. ### Testing * https://torchci-git-fork-huydhn-migrate-llm-dashboard-v3-fbopensource.vercel.app/benchmark/llms?repoName=pytorch%2Fpytorch * https://torchci-git-fork-huydhn-migrate-llm-dashboard-v3-fbopensource.vercel.app/benchmark/llms?repoName=pytorch%2Fexecutorch
- Loading branch information
Showing
9 changed files
with
193 additions
and
135 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
84 changes: 51 additions & 33 deletions
84
torchci/clickhouse_queries/oss_ci_benchmark_branches/query.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,47 +1,65 @@ | ||
| -- This query is used to get the list of branches and commits used by different | ||
| -- OSS CI benchmark experiments. This powers HUD benchmarks dashboards | ||
| WITH benchmarks AS ( | ||
| SELECT | ||
| o.head_branch AS head_branch, | ||
| o.head_sha AS head_sha, | ||
| o.workflow_id AS id, | ||
| IF( | ||
| empty(o.runners), | ||
| tupleElement(o.benchmark, 'extra_info') [ 'device' ], | ||
| tupleElement(o.runners [ 1 ], 'name') | ||
| ) AS device, | ||
| IF( | ||
| empty(o.runners), | ||
| tupleElement(o.benchmark, 'extra_info') [ 'arch' ], | ||
| tupleElement(o.runners [ 1 ], 'type') | ||
| ) AS arch, | ||
| toStartOfDay(fromUnixTimestamp(o.timestamp)) AS event_time | ||
| FROM | ||
| benchmark.oss_ci_benchmark_v3 o | ||
| WHERE | ||
| o.timestamp >= toUnixTimestamp({startTime: DateTime64(3) }) | ||
| AND o.timestamp < toUnixTimestamp({stopTime: DateTime64(3) }) | ||
| AND o.repo = {repo: String } | ||
| AND ( | ||
| has({benchmarks: Array(String) }, o.benchmark.name) | ||
| OR empty({benchmarks: Array(String) }) | ||
| ) | ||
| AND ( | ||
| has({models: Array(String) }, o.model.name) | ||
| OR empty({models: Array(String) }) | ||
| ) | ||
| AND ( | ||
| has({dtypes: Array(String) }, o.benchmark.dtype) | ||
| OR empty({dtypes: Array(String) }) | ||
| ) | ||
| AND ( | ||
| NOT has({excludedMetrics: Array(String) }, o.metric.name) | ||
| OR empty({excludedMetrics: Array(String) }) | ||
| ) | ||
| AND notEmpty(o.metric.name) | ||
| AND notEmpty(o.benchmark.dtype) | ||
| ) | ||
| SELECT | ||
| DISTINCT w.head_branch AS head_branch, | ||
| w.head_sha, | ||
| w.id, | ||
| toStartOfDay(fromUnixTimestamp64Milli(o.timestamp)) AS event_time, | ||
| o.filename | ||
| DISTINCT replaceOne(head_branch, 'refs/heads/', '') AS head_branch, | ||
| head_sha, | ||
| id, | ||
| event_time | ||
| FROM | ||
| benchmark.oss_ci_benchmark_v2 o | ||
| LEFT JOIN default .workflow_run w FINAL ON o.workflow_id = w.id | ||
| benchmarks | ||
| WHERE | ||
| o.timestamp >= toUnixTimestamp64Milli({startTime: DateTime64(3) }) | ||
| AND o.timestamp < toUnixTimestamp64Milli({stopTime: DateTime64(3) }) | ||
| AND ( | ||
| has({filenames: Array(String) }, o.filename) | ||
| OR empty({filenames: Array(String) }) | ||
| ) | ||
| AND ( | ||
| has({names: Array(String) }, o.name) | ||
| OR empty({names: Array(String) }) | ||
| ) | ||
| -- NB: DEVICE (ARCH) is the display format used by HUD when grouping together these two fields | ||
| AND ( | ||
| ( | ||
| CONCAT( | ||
| o.device, | ||
| device, | ||
| ' (', | ||
| IF(empty(o.arch), 'NVIDIA A100-SXM4-40GB', o.arch), | ||
| IF(empty(arch), 'NVIDIA A100-SXM4-40GB', arch), | ||
| ')' | ||
| ) = {deviceArch: String } | ||
| OR {deviceArch: String } = '' | ||
| ) | ||
| AND ( | ||
| has({dtypes: Array(String) }, o.dtype) | ||
| OR empty({dtypes: Array(String) }) | ||
| ) | ||
| AND ( | ||
| NOT has({excludedMetrics: Array(String) }, o.metric) | ||
| OR empty({excludedMetrics: Array(String) }) | ||
| ) | ||
| AND notEmpty(o.metric) | ||
| AND w.html_url LIKE CONCAT('%', {repo: String }, '%') | ||
| AND notEmpty(o.dtype) | ||
| AND notEmpty(o.device) | ||
| AND notEmpty(device) | ||
| ORDER BY | ||
| w.head_branch, | ||
| head_branch, | ||
| event_time DESC |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
119 changes: 73 additions & 46 deletions
119
torchci/clickhouse_queries/oss_ci_benchmark_llms/query.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
97 changes: 58 additions & 39 deletions
97
torchci/clickhouse_queries/oss_ci_benchmark_names/query.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,51 +1,70 @@ | ||
| --- This query is used by HUD benchmarks dashboards to get the list of experiment names | ||
| WITH benchmarks AS ( | ||
| SELECT | ||
| o.benchmark.name AS benchmark, | ||
| o.model.name AS model, | ||
| o.model.backend AS backend, | ||
| o.metric.name AS metric, | ||
| o.benchmark.dtype AS dtype, | ||
| IF( | ||
| empty(o.runners), | ||
| tupleElement(o.benchmark, 'extra_info') [ 'device' ], | ||
| tupleElement(o.runners [ 1 ], 'name') | ||
| ) AS device, | ||
| IF( | ||
| empty(o.runners), | ||
| tupleElement(o.benchmark, 'extra_info') [ 'arch' ], | ||
| tupleElement(o.runners [ 1 ], 'type') | ||
| ) AS arch | ||
| FROM | ||
| benchmark.oss_ci_benchmark_v3 o | ||
| WHERE | ||
| o.timestamp >= toUnixTimestamp({startTime: DateTime64(3) }) | ||
| AND o.timestamp < toUnixTimestamp({stopTime: DateTime64(3) }) | ||
| AND o.repo = {repo: String } | ||
| AND ( | ||
| has({benchmarks: Array(String) }, o.benchmark.name) | ||
| OR empty({benchmarks: Array(String) }) | ||
| ) | ||
| AND ( | ||
| has({models: Array(String) }, o.model.name) | ||
| OR empty({models: Array(String) }) | ||
| ) | ||
| AND ( | ||
| has({dtypes: Array(String) }, o.benchmark.dtype) | ||
| OR empty({dtypes: Array(String) }) | ||
| ) | ||
| AND ( | ||
| NOT has({excludedMetrics: Array(String) }, o.metric.name) | ||
| OR empty({excludedMetrics: Array(String) }) | ||
| ) | ||
| AND notEmpty(o.metric.name) | ||
| AND notEmpty(o.benchmark.dtype) | ||
| ) | ||
| SELECT | ||
| DISTINCT o.filename AS filename, | ||
| o.name, | ||
| o.metric, | ||
| o.dtype, | ||
| o.device, | ||
| -- NB: Default to NVIDIA A100-SXM4-40GB for old records without arch column | ||
| IF(empty(o.arch), 'NVIDIA A100-SXM4-40GB', o.arch) AS arch | ||
| DISTINCT benchmark, | ||
| CONCAT(model, ' ', backend) AS name, | ||
| metric, | ||
| dtype, | ||
| device, | ||
| arch | ||
| FROM | ||
| benchmark.oss_ci_benchmark_v2 o | ||
| LEFT JOIN default .workflow_run w FINAL ON o.workflow_id = w.id | ||
| benchmarks | ||
| WHERE | ||
| o.timestamp >= toUnixTimestamp64Milli({startTime: DateTime64(3) }) | ||
| AND o.timestamp < toUnixTimestamp64Milli({stopTime: DateTime64(3) }) | ||
| AND ( | ||
| has({filenames: Array(String) }, o.filename) | ||
| OR empty({filenames: Array(String) }) | ||
| ) | ||
| AND ( | ||
| has({names: Array(String) }, o.name) | ||
| OR empty({names: Array(String) }) | ||
| ) | ||
| -- NB: DEVICE (ARCH) is the display format used by HUD when grouping together these two fields | ||
| AND ( | ||
| ( | ||
| CONCAT( | ||
| o.device, | ||
| device, | ||
| ' (', | ||
| IF(empty(o.arch), 'NVIDIA A100-SXM4-40GB', o.arch), | ||
| IF(empty(arch), 'NVIDIA A100-SXM4-40GB', arch), | ||
| ')' | ||
| ) = {deviceArch: String } | ||
| OR {deviceArch: String } = '' | ||
| ) | ||
| AND ( | ||
| has({dtypes: Array(String) }, o.dtype) | ||
| OR empty({dtypes: Array(String) }) | ||
| ) | ||
| AND ( | ||
| NOT has({excludedMetrics: Array(String) }, o.metric) | ||
| OR empty({excludedMetrics: Array(String) }) | ||
| ) | ||
| AND notEmpty(o.metric) | ||
| AND w.html_url LIKE CONCAT('%', {repo: String }, '%') | ||
| AND notEmpty(o.dtype) | ||
| AND notEmpty(o.device) | ||
| AND notEmpty(device) | ||
| ORDER BY | ||
| o.filename, | ||
| o.name, | ||
| o.metric, | ||
| o.dtype, | ||
| o.device | ||
| benchmark, | ||
| name, | ||
| metric, | ||
| dtype, | ||
| device |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.