Skip to content

Commit

Permalink
Count models with 0 speedup as failures (#4681)
Browse files Browse the repository at this point in the history
Reported by @xmfan where the dashboard counts models with 0 speedup
(failed perf benchmark) as success. They should be count as failures
instead.

### Testing

https://torchci-git-fork-huydhn-failed-benchmark-models-fbopensource.vercel.app/benchmark/compilers
  • Loading branch information
huydhn authored Oct 27, 2023
1 parent bece5dc commit 54ee5e7
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion torchci/pages/benchmark/compilers.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ function computePassrate(
const workflowId = record.workflow_id;
const suite = record.suite;
const model = record.name;
const accuracy = record.accuracy;

// Use clear compiler name to avoid confusion about what they do
const compiler =
Expand Down Expand Up @@ -210,7 +211,14 @@ function computePassrate(
passCount[bucket][workflowId][suite][compiler] = 0;
}

if (isPass(bucket, workflowId, suite, compiler, model, passingModels)) {
// If the model pass accuracy check but fails the performance benchmark with an
// 0 speedup, it should be counted as a failure. However, `pass_due_to_skip` is
// an exception and it's ok to have 0 speedup there
if (
(isPass(bucket, workflowId, suite, compiler, model, passingModels) &&
record.speedup !== 0.0) ||
accuracy === "pass_due_to_skip"
) {
passCount[bucket][workflowId][suite][compiler] += 1;
}

Expand Down

1 comment on commit 54ee5e7

@vercel
Copy link

@vercel vercel bot commented on 54ee5e7 Oct 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.