Skip to content

[Perf] Linux/x64: 7 Regressions on 5/12/2023 4:02:48 AM #87180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
performanceautofiler bot opened this issue May 16, 2023 · 4 comments
Closed

[Perf] Linux/x64: 7 Regressions on 5/12/2023 4:02:48 AM #87180

performanceautofiler bot opened this issue May 16, 2023 · 4 comments
Assignees
@DrewScoggins
Labels
arch-x64 area-Infrastructure os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented May 16, 2023 •
edited by kunalspathak
Loading

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3a91c9dc96bdfd8164825eec94660691b661f676
Compare f1819bd78f0e7506b8c3c7f90b47aa41b0e22c6f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Repeat - Duration of single invocation 346.23 ns 371.33 ns 1.07 0.11 False
Skip_One - Duration of single invocation 780.23 ns 845.24 ns 1.08 0.02 False
SelectToList - Duration of single invocation 348.43 ns 373.79 ns 1.07 0.10 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Repeat

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 371.3314751054356 > 346.57633689272205.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 4/12/2023 4:44:56 PM, 5/11/2023 9:15:05 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -27.044626631475996 (T) = (0 -371.6218901026913) / Math.Sqrt((54.16111273921081 / (27)) + (0.6695927993724577 / (13))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (13) - 2, .025) and -0.11655383773848124 = (332.82935183438275 - 371.6218901026913) / 332.82935183438275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Skip_One(input: IEnumerable)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 845.2363213337051 > 819.8665655506046.
IsChangePoint: Marked as a change because one of 3/3/2023 4:59:02 PM, 4/12/2023 4:44:56 PM, 5/11/2023 9:15:05 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -12.452415392072043 (T) = (0 -867.5903618379875) / Math.Sqrt((288.3529693506853 / (27)) + (460.44159878628176 / (13))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (13) - 2, .025) and -0.10797174898673698 = (783.0437577775939 - 867.5903618379875) / 783.0437577775939 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Linq.Tests.Perf_Enumerable.SelectToList(input: IList)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 373.78905325858165 > 367.5679942950501.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/11/2023 5:02:01 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -7.427264210953988 (T) = (0 -402.4630251223733) / Math.Sqrt((382.6506143334529 / (26)) + (173.95264067927567 / (14))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (14) - 2, .025) and -0.10637249323779682 = (363.768104849178 - 402.4630251223733) / 363.768104849178 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3a91c9dc96bdfd8164825eec94660691b661f676
Compare f1819bd78f0e7506b8c3c7f90b47aa41b0e22c6f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ProducerConsumer - Duration of single invocation 500.16 ns 617.77 ns 1.24 0.37 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 617.771502375 > 511.41413249062504.
IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 5/11/2023 9:15:05 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -22.845383797738283 (T) = (0 -595.9178274141925) / Math.Sqrt((317.61130051489266 / (26)) + (172.00567406192374 / (13))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (13) - 2, .025) and -0.239754167677755 = (480.67418763385626 - 595.9178274141925) / 480.67418763385626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels May 16, 2023
@EgorBo
Copy link
Member

EgorBo commented May 16, 2023

#86109 likely invalidated PGO data

@EgorBo EgorBo removed the untriaged New issue has not been triaged by the area owner label May 16, 2023
@EgorBo EgorBo mentioned this issue May 16, 2023
Closed
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 6, 2023
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues Jun 6, 2023
@ghost ghost added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 6, 2023
@kunalspathak
Copy link
Member

@ghost
Copy link

ghost commented Jun 8, 2023

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3a91c9dc96bdfd8164825eec94660691b661f676
Compare f1819bd78f0e7506b8c3c7f90b47aa41b0e22c6f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_Enumerable

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Repeat - Duration of single invocation 346.23 ns 371.33 ns 1.07 0.11 False
Skip_One - Duration of single invocation 780.23 ns 845.24 ns 1.08 0.02 False
SelectToList - Duration of single invocation 348.43 ns 373.79 ns 1.07 0.10 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_Enumerable*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_Enumerable.Repeat

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 371.3314751054356 > 346.57633689272205.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 4/12/2023 4:44:56 PM, 5/11/2023 9:15:05 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -27.044626631475996 (T) = (0 -371.6218901026913) / Math.Sqrt((54.16111273921081 / (27)) + (0.6695927993724577 / (13))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (13) - 2, .025) and -0.11655383773848124 = (332.82935183438275 - 371.6218901026913) / 332.82935183438275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Linq.Tests.Perf_Enumerable.Skip_One(input: IEnumerable)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 845.2363213337051 > 819.8665655506046.
IsChangePoint: Marked as a change because one of 3/3/2023 4:59:02 PM, 4/12/2023 4:44:56 PM, 5/11/2023 9:15:05 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -12.452415392072043 (T) = (0 -867.5903618379875) / Math.Sqrt((288.3529693506853 / (27)) + (460.44159878628176 / (13))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (13) - 2, .025) and -0.10797174898673698 = (783.0437577775939 - 867.5903618379875) / 783.0437577775939 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Linq.Tests.Perf_Enumerable.SelectToList(input: IList)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 373.78905325858165 > 367.5679942950501.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/2/2023 3:18:41 PM, 5/11/2023 5:02:01 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -7.427264210953988 (T) = (0 -402.4630251223733) / Math.Sqrt((382.6506143334529 / (26)) + (173.95264067927567 / (14))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (14) - 2, .025) and -0.10637249323779682 = (363.768104849178 - 402.4630251223733) / 363.768104849178 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 3a91c9dc96bdfd8164825eec94660691b661f676
Compare f1819bd78f0e7506b8c3c7f90b47aa41b0e22c6f
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ProducerConsumer - Duration of single invocation 500.16 ns 617.77 ns 1.24 0.37 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 617.771502375 > 511.41413249062504.
IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 5/11/2023 9:15:05 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -22.845383797738283 (T) = (0 -595.9178274141925) / Math.Sqrt((317.61130051489266 / (26)) + (172.00567406192374 / (13))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (13) - 2, .025) and -0.239754167677755 = (480.67418763385626 - 595.9178274141925) / 480.67418763385626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: DrewScoggins
Labels:

os-linux, arch-x64, area-Infrastructure, untriaged, runtime-coreclr, PGO, needs-area-label
Milestone: -

@vcsjones vcsjones removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jul 2, 2023
@DrewScoggins
Copy link
Member

Did we ever get the to the bottom of this? Looking at the charts, the ArrayPool test is still regressed.

@DrewScoggins DrewScoggins mentioned this issue Sep 5, 2023
Closed
@DrewScoggins DrewScoggins mentioned this issue Sep 15, 2023
Closed
@agocke agocke closed this as not planned Won't fix, can't repro, duplicate, stale Jun 18, 2024
@dotnet-policy-service dotnet-policy-service bot removed the untriaged New issue has not been triaged by the area owner label Jun 18, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Jul 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@DrewScoggins DrewScoggins

Labels
arch-x64 area-Infrastructure os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@vcsjones @agocke @EgorBo @DrewScoggins @jeffschwMSFT @kunalspathak