The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2720371	dotnet-runtime	Run performance job script	Log
1055356	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116157
1055372	dotnet/runtime	AOT compile CoreCLR tests	Log
2720117	dotnet-performance	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2720104	dotnet-performance	Send job to Helix (Windows)	Log
2720095	dotnet-runtime	Run performance job script	Log
2719997	dotnet-dotnet	🛡 Component Governance	Log
1055157	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116102
1055065	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116070
1055109	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116145
1055116	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116148
1055104	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116144
2719858	dotnet-runtime	Send ZipArchiveFuzzer to OneFuzz	Log
1054366	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116105
1054890	dotnet/runtime	Send to Helix	Log
1055016	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116109
1054864	dotnet/runtime	AOT compile CoreCLR tests	Log
1054718	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116050
1054693	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116124
1054686	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116123
1054677	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115947
1054683	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116111
2719657	dotnet-runtime	Run performance job script	Log
1054665	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116121
1054617	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115954
1054594	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#114172
1054229	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115983
1054590	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116120
1054507	dotnet/runtime	Send to Helix	Log
1054585	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115914
1054574	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116017
1054569	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113956
1054539	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116010
1054554	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116042
1054547	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116117
1054536	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116102
1054528	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116080
1054286	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116106
1054519	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115315
1054515	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116116
1054323	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116107
1054509	dotnet/runtime	Send to Helix	Log
1054498	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116115
1054465	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116114
1054469	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115974
1054489	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116111
1054477	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115339
1054457	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116070
2719546	dotnet-runtime	Send job to Helix (Unix)	Log
2719539	dotnet-performance	Send job to Helix (Unix)	Log
1054427	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116113
1054434	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115802
1054437	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116104
1054445	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#115856
1054153	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#116097
1054411	dotnet/runtime	Send tests to Helix (Windows)	Log
1054398	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#105654
1054392	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116110
1054410	dotnet/runtime	AOT compile CoreCLR tests	Log
1054133	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115659
1054334	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115746
1054348	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116109
1054320	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116031
1054314	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116099
1054310	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116085
1054204	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116084
1054306	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116082
1054305	dotnet/runtime	Send tests to Helix (Windows)	Log
1054252	dotnet/runtime	Checkout dotnet/runtime@refs/pull/116046/merge to s	Log	dotnet/runtime#116046
2719284	dotnet-dotnet-buildtools-prereqs-docker	🟪 Generate SBOMs	Log
2719436	dotnet-efcore	Build	Log
2719411	dotnet-runtime	Run performance job script	Log
1054031	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116092
2719331	dotnet-dotnet	SDL Binary Analysis Checks (1ES PT)	Log
2719305	dotnet-dotnet	Build (centOSStreamContainer)	Log
2719292	dotnet-runtime	Run performance job script	Log
1054044	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#115609
2719269	dotnet-dotnet-buildtools-prereqs-docker	🟪 Pull Image Builder	Log
1054032	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116092
1053886	dotnet/runtime	Send to Helix	Log
1054005	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116091
1053993	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116010
2719185	dotnet-performance	Send job to Helix (Unix)	Log
2719179	dotnet-runtime	Run performance job script	Log
1053867	dotnet/runtime	AOT compile CoreCLR tests	Log
1053673	dotnet/roslyn	Run Integration Tests	Log
2718928	dotnet-runtime	Run performance job script	Log
2718827	dotnet-dotnet	Build (centOSStreamContainer)	Log
1053466	dotnet/roslyn	Run Integration Tests	Log
1053372	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#116074
2718725	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2718683	dotnet-performance	Component Detection	Log
1053338	dotnet/runtime	Send job to Helix (Unix)	Log
2718689	dotnet-runtime	Run performance job script	Log
2718662	dotnet-dotnet	Compare signing	Log
2718652	dotnet-runtime	Run performance job script	Log
1053188	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116075
1053167	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#116074
1053131	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#115766
1053137	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#115967
Displaying 100 of 1054 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
16	229	1054

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)