slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1015708	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#114438
1015697	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#114548
1014996	dotnet/performance	Performance micro windows 22H2 x64 Open 8.0	Log
2686269	dotnet-runtime	Libraries SuperPMI collection libraries_tests_no_tiered_compilation Checked coreclr osx arm64 Release	Log
2685926	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 9.0	Log
2685916	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1014621	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono_RuntimeTests	Log
2685368	dotnet-sdk	TestBuild: macOS (x64)	Log	#48626
1014197	dotnet/runtime	osx-x64 Debug PALTests	Log
1014135	dotnet/runtime	osx-arm64 Debug PALTests	Log
1014038	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114577
1014017	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114576
1014026	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2685339	dotnet-sdk	TestBuild: macOS (x64)	Log	#48918
2685332	dotnet-sdk	AoT: macOS (x64)	Log	#49129
1013955	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#114462
1013894	dotnet/arcade	Windows_NT Build_Release	Log	dotnet/arcade#15732
1013976	dotnet/runtime	osx-x64 Release NativeAOT	Log
1013967	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1013926	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#114567
1013875	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#114558
1013909	dotnet/runtime	osx-arm64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1013908	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1013846	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#113907
1013801	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114474
1013792	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114364
1013788	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114296
1013772	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114454
1013775	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114441
1013763	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114457
1013786	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#114318
1013784	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
1013779	dotnet/runtime	osx-arm64 Debug PALTests	Log
1013746	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2685037	dotnet-sdk	Darwin_AoT_Tests	Log	#49127
2684953	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 8.0	Log
2684796	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1013114	dotnet/performance	Performance micro windows 22H2 x86 Open 8.0	Log	dotnet/performance#4821
1012821	dotnet/arcade	Linux Build_Debug	Log	dotnet/arcade#15727
1012791	dotnet/performance	Performance roslyn ubuntu 2204 x64 Open main	Log	dotnet/performance#4821
2684434	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2684422	dotnet-sdk	TestBuild: macOS (x64)	Log	#48611
2684344	dotnet-dotnet	Compare Signing - Mac (Packages)	Log
1012375	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#113313
1012366	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#114498
2684178	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1011954	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114481
1011941	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114326
1011921	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log
1011898	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114477
1011871	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log
1011853	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114432
1011850	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#114432
1011844	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#113876
1011726	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#114137
2683663	dotnet-sdk	Darwin_AoT_Tests	Log	#48907
2683662	dotnet-sdk	Darwin	Log	#48906
2683661	dotnet-sdk	Darwin	Log	#48905
2683475	dotnet-sdk	AoT: macOS (x64)	Log	#48610
2683473	dotnet-sdk	TestBuild: macOS (x64)	Log	#48626
1011329	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1011234	dotnet/performance	Performance akadeindexedset windows 22H2 x64 Open main	Log
1011165	dotnet/runtime	osx-arm64 Debug PALTests	Log
2682940	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2682900	dotnet-sdk	Darwin_AoT_Tests	Log	#48906
2682901	dotnet-sdk	Darwin	Log	#48905
1010110	dotnet/runtime	osx-arm64 Debug PALTests	Log
2682616	dotnet-sdk	Darwin	Log	#48907
2682506	dotnet-runtime	Upload Build Artifact BrowserWasm|Log
1009680	dotnet/runtime	osx-arm64 Debug PALTests	Log	dotnet/runtime#114400
1009412	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#114357
2682014	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2681766	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2681751	dotnet-sdk	TestBuild: macOS (x64)	Log	#48610
2681720	dotnet-sdk	Darwin	Log	#48905
2681719	dotnet-sdk	Darwin	Log	#48906
2681479	dotnet-runtime	osx-x64 Release Libraries_CheckedCoreCLR	Log
1007776	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#61318
2680817	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2680566	dotnet-runtime	Libraries SuperPMI collection libraries_tests Checked coreclr osx arm64 Release	Log
2680496	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2680051	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2679959	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1006363	dotnet/performance	Performance micro ubuntu 2204 x64 Open 8.0	Log	dotnet/performance#4816
2679704	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere main	Log
1005361	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#61318
1004695	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#114187
2678845	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1004743	dotnet/runtime	osx-arm64 Release NativeAOT	Log
1004611	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2678803	dotnet-efcore	macOS	Log
2678694	dotnet-razor	macOS release	Log
1004454	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#114233
2678698	dotnet-runtime	iossimulator-x64 release Mono	Log
2678692	dotnet-runtime	tvossimulator-arm64 release Mono	Log
2678690	dotnet-runtime	tvossimulator-arm64 release Mono	Log
2678683	dotnet-aspnetcore	Build: macOS arm64	Log
2678678	dotnet-runtime	iossimulator-x64 release Mono	Log
2678677	dotnet-runtime	iossimulator-arm64 release Mono	Log
2678676	dotnet-runtime	iossimulator-x64 release Mono	Log
Displaying 100 of 288 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	77	288

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.