Android emulator not booting completely on Helix queue

[akoeplinger]

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=472093

Build leg reported

android-x86 Release AllSubsets_Mono

Pull Request

dotnet/runtime#93220

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "Did not detect boot completion variable on device",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=472093
Error message validated: Did not detect boot completion variable on device
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/20/2023 10:43:09 AM UTC

Report

Build	Definition	Test	Pull Request
1047359	dotnet/runtime	Android.Device_Emulator.JIT.Static.Test.WorkItemExecution
1043422	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution
1043374	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
1040002	dotnet/runtime	System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution
1038074	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution
1037237	dotnet/runtime	Android.Device_Emulator.JIT.Static.Test.WorkItemExecution
1037199	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1036170	dotnet/runtime	System.Runtime.Numerics.Tests.WorkItemExecution
1035973	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution
1035844	dotnet/runtime	System.Text.RegularExpressions.Tests.RegexMatchTests.Match_TestThatTimeoutHappens	dotnet/runtime#115369
1034997	dotnet/runtime	System.Private.Xml.Tests.WorkItemExecution
1029552	dotnet/runtime	System.Numerics.Tensors.Net8.Tests.WorkItemExecution	dotnet/runtime#114148
1027363	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1027038	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
1026668	dotnet/runtime	System.Collections.Immutable.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	15

[premun]

fyi @dougbu this seems to be catching cases when Android emulators are not booted properly

[akoeplinger]

Not sure why the result validation doesn't match, do we need to set up something special to monitor the runtime-extra-platforms pipeline?

[dougbu]

This feels very similar to #1383 and #1415. The general theme is the emulator isn't starting as quickly as expected (there's a 5 minute loop checking for boot_completed in the XHarness case) or just isn't started. We haven't made much progress on either issue, partially because only @premun knows much about the emulators and he's busy elsewhere.

the ubuntu.2204.amd64.android.29.open queue is one of many we've had problems with when deploying in our staging environment.

I can see how dotnet/xharness#1106 could help here and suggest we keep an eye on this issue for additional hits.

[akoeplinger]

#1383 should be different since that is about Android devices i.e. there's no emulator to start so if they report not booted the device is usually hosed.

If the emulator issue is really about not starting fast enough I think I'd be happy if you add a sleep 5min into the VM provisioning as a quick workaround.

[AlitzelMendez]

Not sure why the result validation doesn't match, do we need to set up something special to monitor the runtime-extra-platforms pipeline?

for this particular question the problem is not the runtime-extra-platforms (we are analyzing it), it is a problem on our side when there are helix work items internal retries, we are not analyzing the logs of all the attempts, created an issue for this: #1467

[dougbu]

I'd be happy if you add a sleep 5min into the VM provisioning as a quick workaround.

there's a 5 minute loop just prior to the failing sys.boot_completed search in the function starting at https://github.com/dotnet/xharness/blob/38841f0f33ca713ca5d6388c681bdd911425b488/src/Microsoft.DotNet.XHarness.Android/AdbRunner.cs#L191

personally, I'm nervous about adding Thread.Sleep(...) in that code b/c @premun seemed confident my similar actions for #1415 (where I extended a loop searching for a different readiness signal) were unhelpful. we found that "fix" only reduced the likelihood of our validation failures; a build soon after my fix went in failed again and we (temporarily❔) gave up

if someone understands dotnet/xharness better, please chime in❕

[dougbu]

@akoeplinger how are things going w/ your fix attempt(s)❔

[akoeplinger]

I just came back from vacation, will take another stab at this early next week :)

[ivanpovazan]

@ilyas1974 unfortunately, it seems like we are getting more of these failures. Just wanted to check if we can assist somehow in getting it resolved?

[ilyas1974]

I believe Meghna was working on this. @meghnave do you have a current status?

[meghnave]

Hi, I'm not actively working on this - I'm on FR and Ops last week and this week. The week before, I looked into this issue, which is related, as I'm trying to resolve validation errors there likely caused by the emulator not booting. That issue has been around a while and remains a bit of a black box. I'll share any updates as they come.