Skip to content

when all builds fail, BDN should stop after printing first error #1672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 31, 2021
Merged

when all builds fail, BDN should stop after printing first error #1672

merged 1 commit into from
Mar 31, 2021

Conversation

adamsitnik
Copy link
Member

We have the possibility to ask BDN to stop on the first error: --stopOnFirstError true.

The problem is when the users don't ask for that, try to run n benchmarks, all of them fail to build, and BDN prints the same build error n times.

This PR changes that, so when all the builds fail, we stop after printing the first error.

Example:

dotnet run -c Release -f net5.0 --filter *Algo* *IntroGenericTypeArguments* --runtimes corert50 --buildTimeout 5

Before:

PS C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples> dotnet run -c Release -f net5.0 --filter *Algo* *IntroGenericTypeArguments* --runtimes corert50 --buildTimeout 5                                                                                                C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\Attributes\Filters\OperatingSystemsArchitectureFilterAttribute.cs(15,96): warning CS1573: Parameter 'architectures' has no matching param tag in the XML comment for 'OperatingSystemsArchitectureFilterAttribute.OperatingSystemsArchitectureFilterAttribute(bool, params Architecture[])' (but other parameters do) [C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj]
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 4 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet restore -r win-x64 /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\net5.0\54b73874-ae33-4947-89f5-ff747ccb4397
// command took more that the timeout: 5s. Killing the process tree!
// ***** Done, took 00:00:05 (5.24 sec)   *****
// Found 2 benchmarks:
//   Algo_Md5VsSha256.Md5: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
//   Algo_Md5VsSha256.Sha256: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// Build Error: Standard output:
   Restored C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\BenchmarkDotNet.Samples.csproj (in 457 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Diagnostics.Windows\BenchmarkDotNet.Diagnostics.Windows.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x86\BenchmarkDotNet.Disassembler.x86.csproj (in 260 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Annotations\BenchmarkDotNet.Annotations.csproj (in 271 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x64\BenchmarkDotNet.Disassembler.x64.csproj (in 260 ms).
  Determining projects to restore...
 Standard error:
 The configured timeout 00:00:05 was reached!

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\net5.0\54b73874-ae33-4947-89f5-ff747ccb4397
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html

// Build Error: Standard output:
   Restored C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\BenchmarkDotNet.Samples.csproj (in 457 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Diagnostics.Windows\BenchmarkDotNet.Diagnostics.Windows.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x86\BenchmarkDotNet.Disassembler.x86.csproj (in 260 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Annotations\BenchmarkDotNet.Annotations.csproj (in 271 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x64\BenchmarkDotNet.Disassembler.x64.csproj (in 260 ms).
  Determining projects to restore...
 Standard error:
 The configured timeout 00:00:05 was reached!

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\net5.0\54b73874-ae33-4947-89f5-ff747ccb4397
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.Algo_Md5VsSha256-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.Algo_Md5VsSha256-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.Algo_Md5VsSha256-report.html

// * Detailed results *
Algo_Md5VsSha256.Md5: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
Runtime = ; GC =
There are not any results runs

Algo_Md5VsSha256.Sha256: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
Runtime = ; GC =
There are not any results runs

// * Summary *

BenchmarkDotNet=v0.12.1.20210316-develop, OS=Windows 10.0.18363.1440 (1909/November2019Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.100-preview.1.21103.13
  [Host] : .NET 5.0.4 (5.0.421.11614), X64 RyuJIT

Runtime=CoreRT 5.0  Toolchain=Core RT 6.0.0-*

| Method | Mean | Error | Ratio | RatioSD |
|------- |-----:|------:|------:|--------:|
|    Md5 |   NA |    NA |     ? |       ? |
| Sha256 |   NA |    NA |     ? |       ? |

Benchmarks with issues:
  Algo_Md5VsSha256.Md5: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
  Algo_Md5VsSha256.Sha256: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// * Warnings *
BaselineCustomAnalyzer
  Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline value is too close to zero.

// * Legends *
  Mean    : Arithmetic mean of all measurements
  Error   : Half of 99.9% confidence interval
  Ratio   : Mean of the ratio distribution ([Current]/[Baseline])
  RatioSD : Standard deviation of the ratio distribution ([Current]/[Baseline])
  1 ns    : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// ** Remained 2 benchmark(s) to run **
Run time: 00:00:02 (2.93 sec), executed benchmarks: 0

// Found 1 benchmarks:
//   IntroGenericTypeArguments<Int32>.Create: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// Build Error: Standard output:
   Restored C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\BenchmarkDotNet.Samples.csproj (in 457 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Diagnostics.Windows\BenchmarkDotNet.Diagnostics.Windows.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x86\BenchmarkDotNet.Disassembler.x86.csproj (in 260 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Annotations\BenchmarkDotNet.Annotations.csproj (in 271 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x64\BenchmarkDotNet.Disassembler.x64.csproj (in 260 ms).
  Determining projects to restore...
 Standard error:
 The configured timeout 00:00:05 was reached!

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\net5.0\54b73874-ae33-4947-89f5-ff747ccb4397
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.IntroGenericTypeArguments_Int32_-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.IntroGenericTypeArguments_Int32_-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.IntroGenericTypeArguments_Int32_-report.html

// * Detailed results *
IntroGenericTypeArguments<Int32>.Create: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
Runtime = ; GC =
There are not any results runs

// * Summary *

BenchmarkDotNet=v0.12.1.20210316-develop, OS=Windows 10.0.18363.1440 (1909/November2019Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.100-preview.1.21103.13
  [Host] : .NET 5.0.4 (5.0.421.11614), X64 RyuJIT

Runtime=CoreRT 5.0  Toolchain=Core RT 6.0.0-*

| Method | Mean | Error |
|------- |-----:|------:|
| Create |   NA |    NA |

Benchmarks with issues:
  IntroGenericTypeArguments<Int32>.Create: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// ** Remained 1 benchmark(s) to run **
Run time: 00:00:00 (0.08 sec), executed benchmarks: 0

// Found 1 benchmarks:
//   IntroGenericTypeArguments<Char>.Create: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// Build Error: Standard output:
   Restored C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\BenchmarkDotNet.Samples.csproj (in 457 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Diagnostics.Windows\BenchmarkDotNet.Diagnostics.Windows.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj (in 398 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x86\BenchmarkDotNet.Disassembler.x86.csproj (in 260 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Annotations\BenchmarkDotNet.Annotations.csproj (in 271 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x64\BenchmarkDotNet.Disassembler.x64.csproj (in 260 ms).
  Determining projects to restore...
 Standard error:
 The configured timeout 00:00:05 was reached!

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\net5.0\54b73874-ae33-4947-89f5-ff747ccb4397
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.IntroGenericTypeArguments_Char_-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.IntroGenericTypeArguments_Char_-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.IntroGenericTypeArguments_Char_-report.html

// * Detailed results *
IntroGenericTypeArguments<Char>.Create: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
Runtime = ; GC =
There are not any results runs

// * Summary *

BenchmarkDotNet=v0.12.1.20210316-develop, OS=Windows 10.0.18363.1440 (1909/November2019Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.100-preview.1.21103.13
  [Host] : .NET 5.0.4 (5.0.421.11614), X64 RyuJIT

Runtime=CoreRT 5.0  Toolchain=Core RT 6.0.0-*

| Method | Mean | Error |
|------- |-----:|------:|
| Create |   NA |    NA |

Benchmarks with issues:
  IntroGenericTypeArguments<Char>.Create: Job-HRQWCB(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// ** Remained 0 benchmark(s) to run **
Run time: 00:00:00 (0.08 sec), executed benchmarks: 0

Global total time: 00:00:08 (8.36 sec), executed benchmarks: 0
// * Artifacts cleanup *

After:

PS C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples> dotnet run -c Release -f net5.0 --filter *Algo* *IntroGenericTypeArguments* --runtimes corert50 --buildTimeout 5                                                                                                C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\Attributes\Filters\OperatingSystemsArchitectureFilterAttribute.cs(15,96): warning CS1573: Parameter 'architectures' has no matching param tag in the XML comment for 'OperatingSystemsArchitectureFilterAttribute.OperatingSystemsArchitectureFilterAttribute(bool, params Architecture[])' (but other parameters do) [C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj]
C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\Attributes\Filters\OperatingSystemsFilterAttribute.cs(29,74): warning CS1573: Parameter 'platforms' has no matching param tag in the XML comment for 'OperatingSystemsFilterAttribute.OperatingSystemsFilterAttribute(bool, params OS[])' (but other parameters do) [C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj]
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 4 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet restore -r win-x64 /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\net5.0\d243adbc-bc40-4401-8d69-16109844ce44
// command took more that the timeout: 5s. Killing the process tree!
// ***** Done, took 00:00:05 (5.22 sec)   *****
// Found 2 benchmarks:
//   Algo_Md5VsSha256.Md5: Job-YUXPVP(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
//   Algo_Md5VsSha256.Sha256: Job-YUXPVP(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// Build Error: Standard output:
   Restored C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\BenchmarkDotNet.Samples.csproj (in 455 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet\BenchmarkDotNet.csproj (in 401 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Diagnostics.Windows\BenchmarkDotNet.Diagnostics.Windows.csproj (in 393 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Annotations\BenchmarkDotNet.Annotations.csproj (in 274 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x64\BenchmarkDotNet.Disassembler.x64.csproj (in 258 ms).
  Restored C:\Projects\BenchmarkDotNet\src\BenchmarkDotNet.Disassembler.x86\BenchmarkDotNet.Disassembler.x86.csproj (in 258 ms).
  Determining projects to restore...
 Standard error:
 The configured timeout 00:00:05 was reached!

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Projects\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\net5.0\d243adbc-bc40-4401-8d69-16109844ce44
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html
// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.Algo_Md5VsSha256-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.Algo_Md5VsSha256-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkDotNet.Samples.Algo_Md5VsSha256-report.html

// * Detailed results *
Algo_Md5VsSha256.Md5: Job-YUXPVP(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)
Runtime = ; GC =
There are not any results runs

// * Summary *

BenchmarkDotNet=v0.12.1.20210316-develop, OS=Windows 10.0.18363.1440 (1909/November2019Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.100-preview.1.21103.13
  [Host] : .NET 5.0.4 (5.0.421.11614), X64 RyuJIT

Runtime=CoreRT 5.0  Toolchain=Core RT 6.0.0-*

| Method | Mean | Error | Ratio | RatioSD |
|------- |-----:|------:|------:|--------:|
|    Md5 |   NA |    NA |     ? |       ? |

Benchmarks with issues:
  Algo_Md5VsSha256.Md5: Job-YUXPVP(Runtime=CoreRT 5.0, Toolchain=Core RT 6.0.0-*)

// * Warnings *
BaselineCustomAnalyzer
  Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline value is too close to zero.

// * Legends *
  Mean    : Arithmetic mean of all measurements
  Error   : Half of 99.9% confidence interval
  Ratio   : Mean of the ratio distribution ([Current]/[Baseline])
  RatioSD : Standard deviation of the ratio distribution ([Current]/[Baseline])
  1 ns    : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// ** Remained 2 benchmark(s) to run **
Run time: 00:00:02 (2.6 sec), executed benchmarks: 0

Global total time: 00:00:07 (7.84 sec), executed benchmarks: 0
// * Artifacts cleanup *

It's very useful for solutions that have a LOT of benchmarks, like the dotnet/performance repo with 3.5k benchmarks.

cc @danmoseley

@adamsitnik adamsitnik added this to the v0.13.0 milestone Mar 16, 2021
AndreyAkinshin
AndreyAkinshin approved these changes Mar 31, 2021
View reviewed changes
Copy link
Member

@AndreyAkinshin AndreyAkinshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adamsitnik
Copy link
Member Author

The CI failures are caused by #1542, I am going to squash it now as is

@adamsitnik adamsitnik merged commit 63e28c1 into master Mar 31, 2021
@adamsitnik adamsitnik deleted the stopOnBuildError branch March 31, 2021 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AndreyAkinshin AndreyAkinshin AndreyAkinshin approved these changes

Assignees
No one assigned
Labels
Area:Toolchains enhancement
Projects
None yet
Milestone
v0.13.0
Development

Successfully merging this pull request may close these issues.
2 participants
@adamsitnik @AndreyAkinshin