Microsoft.Testing.Extensions.HangDump does not work on macOS or Linux in native AOT mode #3097

martincostello · 2024-06-13T16:39:30Z

Describe the bug

Following this tip #3095 (comment) I added the Microsoft.Testing.Extensions.HangDump package to a native AoT test project of mine to try and diagnose a hanging test (which in my case I can only repro on Linux).

Upon making the changes martincostello/alexa-london-travel#1298 and running the CI, the tests fail with a unique exception message each on macOS and Linux as shown below. I can also repo this outside of GitHub Actions CI with WSL.

macOS

Unhandled Exception: System.ArgumentOutOfRangeException: The path '/var/folders/dm/88b38gj92jj53dgxdsm12qf00000gn/T/hangdumpgeneratorpipename.240706229_b5437ec2c2144a679172c9d6fe2a036d/.p' is of an invalid length for use with domain sockets on this platform.  The length must be between 1 and 104 characters, inclusive. (Parameter 'path')
Actual value was /var/folders/dm/88b38gj92jj53dgxdsm12qf00000gn/T/hangdumpgeneratorpipename.240706229_b5437ec2c2144a679172c9d6fe2a036d/.p.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0x710
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0xfd0
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x2d4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x670
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.TestHostControlledHost.<RunAsync>d__4.MoveNext() + 0xec
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Builder.TestApplication.<RunAsync>d__17.MoveNext() + 0xc4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Helpers.TaskExtensions.<>c.<<TimeoutAfterAsync>b__2_0>d.MoveNext() + 0xa8
   at TestingPlatformEntryPoint.<Main>d__0.MoveNext() + 0x284
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at TestingPlatformEntryPoint.<Main>(String[] args) + 0x3c
   at LondonTravel.Skill!<BaseAddress>+0xae7ca8
Unhandled Exception: System.InvalidOperationException: Unexpected state in file '/_/src/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs' at line '185'
   at Microsoft.Testing.Platform.Helpers.ApplicationStateGuard.Ensure(Boolean, String, Int32) + 0xb0
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpProcessLifetimeHandler.<OnTestHostProcessStartedAsync>d__42.MoveNext() + 0x6c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.TestHostControllersTestHost.<InternalRunAsync>d__21.MoveNext() + 0x238c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.TestHostControllersTestHost.<InternalRunAsync>d__21.MoveNext() + 0x2d48
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x2d4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x670
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at Microsoft.Testing.Platform.Builder.TestApplication.<RunAsync>d__17.MoveNext() + 0xc4
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at TestingPlatformEntryPoint.<Main>d__0.MoveNext() + 0x284
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x24
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0x100
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x68
   at TestingPlatformEntryPoint.<Main>(String[] args) + 0x3c
   at LondonTravel.Skill!<BaseAddress>+0xae7ca8

Linux

Unhandled Exception: System.IO.IOException: Broken pipe
 ---> System.Net.Sockets.SocketException (32): Broken pipe
   at System.Exception.SetCurrentStackTrace() + 0x63
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.SetCurrentStackTrace(Exception) + 0x18
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError, Boolean) + 0x72
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsync(Socket, CancellationToken) + 0xe1
   at System.Net.Sockets.Socket.SendAsync(ReadOnlyMemory`1, SocketFlags, CancellationToken) + 0xea
   at System.IO.Pipes.PipeStream.<WriteAsyncCore>d__83.MoveNext() + 0x93
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&) + 0x42
   at System.IO.Pipes.PipeStream.WriteAsyncCore(ReadOnlyMemory`1, CancellationToken) + 0x4a
   at System.IO.Pipes.PipeStream.WriteAsync(ReadOnlyMemory`1, CancellationToken) + 0xb4
   at Microsoft.Testing.Platform.IPC.NamedPipeClient.<RequestReplyAsync>d__13`2.MoveNext() + 0x8fb
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&) + 0x5e
   at Microsoft.Testing.Platform.IPC.NamedPipeClient.RequestReplyAsync[TRequest,TResponse](TRequest, CancellationToken) + 0x63
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpActivityIndicator.<OnTestSessionFinishingAsync>d__33.MoveNext() + 0x13f
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&) + 0x42
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpActivityIndicator.OnTestSessionFinishingAsync(SessionUid, CancellationToken) + 0x35
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<NotifyTestSessionEndAsync>d__12.MoveNext() + 0x176
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task`1.TrySetResult(TResult) + 0x88
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1, TResult) + 0x39
   at Microsoft.Testing.Platform.Messages.MessageBusProxy.<DrainDataAsync>d__6.MoveNext() + 0xef
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task`1.TrySetResult(TResult) + 0x88
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1, TResult) + 0x39
   at Microsoft.Testing.Platform.Messages.AsynchronousMessageBus.<DrainDataAsync>d__17.MoveNext() + 0x8a3
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task`1.TrySetResult(TResult) + 0x86
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1, TResult) + 0x38
   at Microsoft.Testing.Platform.Messages.AsyncConsumerDataProcessor.<DrainDataAsync>d__14.MoveNext() + 0x329
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) + 0x8c
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) + 0x66
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) + 0xa7
   at System.Threading.Tasks.Task.RunContinuations(Object) + 0xa8
   at System.Threading.Tasks.Task.TrySetResult() + 0x6f
   at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut() + 0x13
   at System.Threading.TimerQueueTimer.Fire(Boolean) + 0x60
   at System.Threading.TimerQueue.FireNextTimers() + 0x23b
   at System.Threading.ThreadPoolWorkQueue.Dispatch() + 0x2d1
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() + 0x15a
   at System.Threading.Thread.StartThread(IntPtr) + 0xee
   at System.Threading.Thread.ThreadEntryPoint(IntPtr) + 0x19
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at System.IO.Pipes.PipeStream.<WriteAsyncCore>d__83.MoveNext() + 0x16b
   --- End of inner exception stack trace ---
   at System.IO.Pipes.PipeStream.<WriteAsyncCore>d__83.MoveNext() + 0x23e
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.IPC.NamedPipeClient.<RequestReplyAsync>d__13`2.MoveNext() + 0x94c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at Microsoft.Testing.Platform.Helpers.TaskExtensions.<>c.<<TimeoutAfterAsync>b__2_0>d.MoveNext() + 0x96
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<ExecuteRequestAsync>d__8.MoveNext() + 0x318
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0x7b6
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at Microsoft.Testing.Platform.Hosts.ConsoleTestHost.<InternalRunAsync>d__9.MoveNext() + 0x1094
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x2ae
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at Microsoft.Testing.Platform.Hosts.CommonTestHost.<RunAsync>d__6.MoveNext() + 0x5f3
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Hosts.TestHostControlledHost.<RunAsync>d__4.MoveNext() + 0xd9
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Builder.TestApplication.<RunAsync>d__17.MoveNext() + 0xaf
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at TestingPlatformEntryPoint.<Main>d__0.MoveNext() + 0x244
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at TestingPlatformEntryPoint.<Main>(String[] args) + 0x2e
   at LondonTravel.Skill!<BaseAddress>+0xa10b7c

Steps To Reproduce

Clone martincostello/alexa-london-travel@b93a00a onto a Linux or macOS machine.
Run ./build.ps1 in the root of the repository.

Expected behavior

Either:

The tests pass, or:
The tests produce a dump file relating to a hanging test.

Actual behavior

The process exits with an error of either ArgumentOutOfRangeException on macOS or IOException on Linux.

Additional context

In case it was an issue with the file name I was providing, I tried not specifying a file name at all in the hope of it having a default like dotnet test does. In that case, a different failure is observed:

Unhandled Exception: System.InvalidOperationException: Cannot find mutex 'TESTINGPLATFORM_HANGDUMP_MUTEXNAME_d28475404b504b3aacf951ff5509d4c6'
   at Microsoft.Testing.Extensions.Diagnostics.HangDumpProcessLifetimeHandler.<ActivityTimerAsync>d__44.MoveNext() + 0x39c
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at Microsoft.Testing.Platform.Helpers.SystemTask.<>c__DisplayClass4_0.<RunLongRunning>b__0() + 0x92
   at System.Threading.Thread.StartThread(IntPtr) + 0xee
   at System.Threading.Thread.ThreadEntryPoint(IntPtr) + 0x19

The text was updated successfully, but these errors were encountered:

Evangelink · 2024-06-14T15:40:40Z

@MarcoRossignoli I think it's another bug of pipe name under non-windows, could you check it please?

MarcoRossignoli · 2024-06-14T19:41:00Z

Thanks @martincostello for reporting it, I'll check the length name issue, anyway I don't know if hang/crash works for native aot version.

@brianrob @tommcdon we're using the environment variable for the crash dump https://learn.microsoft.com/en-us/dotnet/core/diagnostics/collect-dumps-crash and the Microsoft.Diagnostics.NETCore.Client to take a dump in case of hang, is it working in native aot mode?

Evangelink · 2024-06-25T10:23:15Z

Sorry I totally forgot about this ticket, will work on it this week!

Evangelink · 2024-06-27T13:39:48Z

@martincostello Would you be able to test the fix from our preview feed? I tried cloning your repo to test it out but running build.ps1 on Ubuntu, I get

  LondonTravel.Skill failed with 2 error(s) (24.3s) → artifacts/bin/LondonTravel.Skill/release_linux-arm64/bootstrap.dll
    clang : error : linker command failed with exit code 1 (use -v to see invocation)
    /home/amaury/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.6/build/Microsoft.NETCore.Native.targets(366,5): error MSB3073: The command ""clang" "/home/amaury/alexa-london-travel/artifacts/obj/LondonTravel.Skill/release_linux-arm64/native/bootstrap.o" -o "/home/amaury/alexa-london-travel/artifacts/bin/LondonTravel.Skill/release_linux-arm64/native/bootstrap" -Wl,--version-script=/home/amaury/alexa-london-travel/artifacts/obj/LondonTravel.Skill/release_linux-arm64/native/bootstrap.exports -Wl,--export-dynamic -gz=zlib -fuse-ld=bfd /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libbootstrapper.o /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libRuntime.WorkstationGC.a /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libeventpipe-disabled.a /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/sdk/libstdc++compat.a /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.Native.a /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.IO.Compression.Native.a /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.Net.Security.Native.a /home/amaury/.nuget/packages/runtime.linux-arm64.microsoft.dotnet.ilcompiler/8.0.6/framework/libSystem.Security.Cryptography.Native.OpenSsl.a --target=aarch64-linux-gnu -g -Wl,-rpath,'$ORIGIN' -Wl,--build-id=sha1 -Wl,--as-needed -pthread -ldl -lz -lrt -lm -pie -Wl,-pie -Wl,-z,relro -Wl,-z,now -Wl,--eh-frame-hdr -Wl,--discard-all -Wl,--gc-sections" exited with code 1.

martincostello · 2024-06-27T13:43:34Z

I can try it out tomorrow, but these are the dependency steps the CI runs to install the tooling that native AoT needs on linux: https://github.com/martincostello/alexa-london-travel/blob/9a799b548274c270f8dc61417265d04e5cc19a9d/.github/workflows/build.yml#L65-L74

Evangelink · 2024-06-27T14:18:47Z

It's ok, I can reproduce the error with these lines :) Thanks @martincostello <3

Evangelink · 2024-06-27T15:22:45Z

We fixed the length issue of the pipe but looks like there is still an issue. We haven't yet validated if hang dump is working well in Native AOT mode so for now the only recommendation would be to not use it for NAOT

We will work on adding tests for it on our next iteration and will post-back the resutls.

martincostello · 2024-06-27T15:26:14Z

the only recommendation would be to not use it for NAOT

The reason I found this bug in the the first place was I was specifically trying to diagnose a hang that is only happening to me in native AoT 😅

MarcoRossignoli · 2024-06-27T15:31:53Z

The reason I found this bug in the the first place was I was specifically trying to diagnose a hang that is only happening to me in native AoT 😅

We're waiting some internal info on how to support hang/crash in native aot mode. We'll let you know soon as possible.

MarcoRossignoli · 2024-06-27T15:32:35Z

The reason I found this bug in the the first place was I was specifically trying to diagnose a hang that is only happening to me in native AoT 😅

@martincostello do you mean that if you run test in "normal mode" everything is good but if you try with native aot it's hanging?

tommcdon · 2024-06-27T19:37:36Z

@brianrob @tommcdon we're using the environment variable for the crash dump https://learn.microsoft.com/en-us/dotnet/core/diagnostics/collect-dumps-crash and the Microsoft.Diagnostics.NETCore.Client to take a dump in case of hang, is it working in native aot mode?

Sorry for the delayed response! Environment variable enabled crash dumps and dotnet-dump adhoc dump collection is possible with NativeAOT if we include a copy of .NET's createdump tool along with the application. The steps needed to collect dumps with NativeAOT are included in dotnet/diagnostics#4150, copy/pasted here:

In .NET 8, using createdump for native AOT applications require some manual steps:

Need the C++ runtime
Need access to the 8.0 .NET core version of createdump. The .NET core version of createdump was modified to work with native AOT applications.

cc @LakshanF @agocke

martincostello · 2024-06-28T05:37:18Z

do you mean that if you run test in "normal mode" everything is good but if you try with native aot it's hanging?

Exactly that.

vitek-karas · 2024-06-28T12:04:07Z

@martincostello - if you publish the tests for AOT do you get any trim/AOT warnings?

martincostello · 2024-06-28T12:10:28Z

Yes, there's a few: https://github.com/martincostello/alexa-london-travel/actions/runs/9710114339/job/26800478379?pr=1298#step:5:208

vitek-karas · 2024-06-28T12:14:00Z

Those are not warnings - just info messages. It seems there are no warnings, unless they're suppressed by something.

Evangelink · 2024-06-28T12:27:12Z

@vitek-karas I haven't seen any warning when building locally. I will double check the configuration to ensure the analyzers are enabled correctly.

agocke · 2024-06-28T17:48:18Z

@martincostello Hi martin, for debugging Native AOT apps in particular, you might find native tooling better. For instance, you could use GDB or LLDB to attach to the running process and dump the stack directly. As long as symbols are on the machine, you shouldn't need any special technology to do basic investigations. The native compiler should be able to understand the native code.

martincostello · 2024-06-29T08:28:21Z

Specifically I only seem to get the hangs in the native AoT tests themselves.

The AoT application itself is fine, and I don't get the hangs in normal xunit tests, or on Windows or macOS under native AoT. It seems to be exclusively my native AoT tests on Linux, and then only sometimes (and those tests are 99% copy-paste from existing non-AoT tests for xunit).

It's mostly just a minor annoyance (a few retries and it'll go away), but I'd like to get to the bottom of it to know if it's an issue in my code, or in the native AoT test SDK itself.

Using the built-in tooling seemed like the easiest way to grab a file to inspect to find what was hanging from my CI (which is where I experience the issue), but that ultimately lead me to opening this issue as I fell at the first hurdle.

agocke · 2024-06-30T14:57:17Z

I assume your Native aot tests are also Native aot? Or are they regular coreclr?

martincostello · 2024-06-30T15:20:53Z

They use the native AoT support in MSTest.TestFramework.

agocke · 2024-07-01T20:38:29Z

Got it. Yup, in that case I would recommend trying to attach with LLDB or GDB. While the dotnet-dump tools have some limited support for Native AOT, the native tools may actually be better in this case.

martincostello · 2024-07-02T08:18:30Z

I might try that, but I'll have to learn LLDB/GDB as I've never used them before. That depends on me being able to reproduce it locally - if it only happens in CI then that's basically a non-starter.

agocke · 2024-07-02T14:33:57Z

Agreed. I’ll leave this issue open. Unfortunately the person who worked on createdump support with Native aot is out on vacation right now, so it might be a bit before we have a more detailed fix.

agocke · 2024-07-03T01:47:23Z

OK, I've investigated and have some more information. There may be multiple ways that the runtime creates dump files (I haven't surveyed all the possible options) but one is a program called createdump that's bundled with the .NET install. That does seem to work to create full dump files.

What I'm unclear on is: how does the Microsoft.Testing.Extensions.HangDump system work? Does it invoke createdump directly?

MarcoRossignoli · 2024-07-03T08:31:11Z

What I'm unclear on is: how does the Microsoft.Testing.Extensions.HangDump system work? Does it invoke createdump directly?

As reported here #3097 (comment) we use the env vars for the crash dump and the package Microsoft.Diagnostics.NETCore.Client to take a dump of the process in case of hang.

agocke · 2024-07-10T21:48:46Z

OK, I managed to investigate the current state and get a dump out of dotnet-dump, which I think means the same thing can be done using the Microsoft.Diagnostics.NETCore.Client library.

Here are the requirements:

The project needs to be built with <EventSourceSupport>true</>. This enables the event pipe that allows another process to talk to the AOT project.
The createdump binary needs to be next to the AOT binary. That lives either in the SDK installation, or in the Microsoft.NETCore.App.runtime.<rid> runtime pack.
createdump needs to be executable.
The process needed to have been started with the environment variable DOTNET_DbgEnableMiniDump=1. I think the NETCore.Client library should handle this.

What I think we need to improve on the Native AOT side is error messages. There's no indication what the cause of some of these failures are.

LakshanF · 2024-07-31T18:01:46Z

Might also need to set DOTNET_DbgMiniDumpType=4 for the instructions above to get the crash dump.

dotnet-dump could be used to collect dumps as well.

lldb , together with the symbol file, can be used to analyze the dumps collected.

martincostello · 2024-08-06T08:52:40Z

Trying to follow the steps above to use dotnet-dump to generate a dump to see where the hang is coming from, and I keep getting the following error:

> dotnet dump collect -p 27454

Writing full to /home/martin/core_20240806_094936
Write dump failed - HRESULT: 0x80004005.

I've tried making every createdump executable and setting the mentioned environment variable, but that hasn't resolved it.

If I try and use dotnet gcdump it just hangs.

martincostello · 2024-08-06T09:48:38Z

Playing around in GitHub Actions, I found a clue that's gotten generation working there.

When I set DOTNET_DbgEnableMiniDump=1 at the job level, I get this warning during publish:

DOTNET_DbgEnableMiniDump is set and the createdump binary does not exist: /home/runner/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.7/tools/createdump

If I manually copy the file from /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.32/createdump (found via find /usr/share/dotnet -name createdump) to the publish directory, a dump is created successfully.

martincostello · 2024-08-06T13:27:42Z

The tips got me far enough that I think I root-caused the deadlock as described in #3485, which I've now been able to create a workaround for in my project. The documentation for trying to do this with a native AoT app certainly needs some TLC for those unfamiliar with tools such as lldb (which is how I eventually got hold of the stacks) 😅.

microsoft-github-policy-service bot added the Needs: Triage 🔍 label Jun 13, 2024

This was referenced Jun 13, 2024

--blame-hang equivalent for AoT testing #3095

Closed

Collect hang dumps from native AoT tests martincostello/alexa-london-travel#1298

Closed

Evangelink added Type: Bug Area: MTP Extensions and removed Needs: Triage 🔍 labels Jun 14, 2024

Evangelink self-assigned this Jun 25, 2024

Evangelink added this to the 3.5.0 milestone Jun 25, 2024

Evangelink mentioned this issue Jun 25, 2024

Shorten pipe names #3183

Merged

MarcoRossignoli changed the title ~~Microsoft.Testing.Extensions.HangDump does not work on macOS or Linux~~ Microsoft.Testing.Extensions.HangDump does not work on macOS or Linux in native AOT mode Jul 2, 2024

MarcoRossignoli added the Type: Feature label Jul 2, 2024

Evangelink removed this from the 3.5.0 milestone Jul 8, 2024

martincostello mentioned this issue Aug 6, 2024

Intermittent deadlock on Linux using native AoT support from console output #3485

Closed

Evangelink removed Type: Feature labels Dec 6, 2024

Microsoft.Testing.Extensions.HangDump does not work on macOS or Linux in native AOT mode #3097

Microsoft.Testing.Extensions.HangDump does not work on macOS or Linux in native AOT mode #3097

Comments

martincostello commented Jun 13, 2024

Describe the bug

macOS

Linux

Steps To Reproduce

Expected behavior

Actual behavior

Additional context

Evangelink commented Jun 14, 2024

Uh oh!

MarcoRossignoli commented Jun 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Evangelink commented Jun 25, 2024

Uh oh!

Evangelink commented Jun 27, 2024

Uh oh!

martincostello commented Jun 27, 2024

Uh oh!

Evangelink commented Jun 27, 2024

Uh oh!

Evangelink commented Jun 27, 2024

Uh oh!

martincostello commented Jun 27, 2024

Uh oh!

MarcoRossignoli commented Jun 27, 2024

Uh oh!

MarcoRossignoli commented Jun 27, 2024

Uh oh!

tommcdon commented Jun 27, 2024

Uh oh!

martincostello commented Jun 28, 2024

Uh oh!

vitek-karas commented Jun 28, 2024

Uh oh!

martincostello commented Jun 28, 2024

Uh oh!

vitek-karas commented Jun 28, 2024

Uh oh!

Evangelink commented Jun 28, 2024

Uh oh!

agocke commented Jun 28, 2024

Uh oh!

martincostello commented Jun 29, 2024

Uh oh!

agocke commented Jun 30, 2024

Uh oh!

martincostello commented Jun 30, 2024

Uh oh!

agocke commented Jul 1, 2024

Uh oh!

martincostello commented Jul 2, 2024

Uh oh!

agocke commented Jul 2, 2024

Uh oh!

agocke commented Jul 3, 2024

Uh oh!

MarcoRossignoli commented Jul 3, 2024

Uh oh!

agocke commented Jul 10, 2024

Uh oh!

LakshanF commented Jul 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martincostello commented Aug 6, 2024

Uh oh!

martincostello commented Aug 6, 2024

Uh oh!

martincostello commented Aug 6, 2024

Uh oh!

MarcoRossignoli commented Jun 14, 2024 •

edited

Loading

LakshanF commented Jul 31, 2024 •

edited

Loading