Intermittent deadlock on Linux using native AoT support from console output #3485
Comments
|
@jkotas do you know who owns |
|
@Evangelink @nohwnd do you recall if some code inside MSTest is doing some wild lock on |
|
@martincostello |
You can find dotnet/rutime area owners in https://github.com/dotnet/runtime/blob/main/docs/area-owners.md However, this does not look like a System.Console bug to me. The stacktraces do not explain why there is a deadlock. It is more likely a bug in native AOT runtime . Have you tried to run latest .NET 9 Preview? It may be a bug that was fixed in .NET 9. |
|
While trying to work out why my Lambda function isn't working when deployed to AWS Lambda with RC.1, I managed to by-accident generate a crash dump on both Linux and macOS. The Linux one showed a deadlock on two threads trying to do a
I can share a link to the crash dump file over email. This was in normal xunit, rather than anything to do with AoT, but the commonality appears to be access to the console. |
|
This is likely duplicate of dotnet/runtime#104340 that will be fixed in .NET 9 RC2. |
|
@martincostello are you still experiencing the issue? Have you been able to test the fix from the linked runtime issue? |
Evangelink
added
the
External: Other
|
I'll revert my workaround in a branch some time this week and see if the problem has resolved. |
Restore logging to native AoT tests. See microsoft/testfx#3485 (comment).
Restore logging to native AoT tests. See microsoft/testfx#3485 (comment).
|
I reverted my workaround and ran the tests on Linux 10 times with no issues, so I assume it's been fixed. |
Describe the bug
Today I've come back to the underlying issue that lead to me opening #3097 to try and work out what's going wrong. I think I've now got a fairly good idea of where the issue is (but not why) and have worked around the issue.
From trial and error in martincostello/alexa-london-travel#1368, I managed to get hold of a dump on Linux under WSL and got the stacks for each thread: martincostello/alexa-london-travel#1368 (comment)
The interesting stacks were the following:
It looks like there's a deadlock between trying to write to the console via
ILoggerin the code under test and the MSTest platform trying to change the colour of the console at the end of the test run before reporting the test summary.It may be that this is an issue in .NET rather than VSTest, but I figured I'd start here in the first instance as I only seem to get this problem using the native AoT test support.
Some interesting points in the code are listed below:
ConsoleDisplayService.DisplayAfterSessionEndRunAsync()sets the console colour just before writing the test summary.SystemConsole.SetForegroundColor()to setConsole.ForegroundColor.Console.ForegroundColorcalls through toConsolePal.RefreshColors().RefreshColors()locks onConsole.Out.SyncTextWriter.Write()- this uses[MethodImpl(MethodImplOptions.Synchronized)]on the methods, so I think that's effectively also locking onConsole.Out.The fact that only the Linux implementation of the setter of
Console.ForegroundColorappears to do any locking would also help explain why I only see this on Linux.I've worked around this issue by reducing the amount of console output from the tests. With the workaround applied, the tests happily keep passing if I run them continuously in a loop (until the job timeout limit of 20 minutes is reached).
However, what I'm at a loss to explain is why there's a deadlock - just that there appears to be one on usage of the console.
Steps To Reproduce
build.ps1Expected behavior
The tests pass and exit successfully.
Actual behavior
The tests deadlock.
The text was updated successfully, but these errors were encountered: