Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault in tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh on Ubuntu. #59758

Closed
danmoseley opened this issue Sep 29, 2021 · 4 comments
Closed

Segfault in tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh on Ubuntu. #59758

danmoseley opened this issue Sep 29, 2021 · 4 comments
Labels
area-Tracing-mono

Comments

@danmoseley
Copy link
Member

Console log: 'PayloadGroup0' from job 032a526a-2272-4191-8743-d045d3f88f1a (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 on ddvsotx2l285

#57324

Note the network disconnect callstack -- I do not know whether that's relevant, or simply a symptom of the other end being cleaned up.

  Discovered:  tracing.eventpipe.XUnitWrapper (found 12 test cases)
  Starting:    tracing.eventpipe.XUnitWrapper (parallel test collections = on, max threads = 4)
   tracing.eventpipe.XUnitWrapper: [Long Running Test] 'tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh', Elapsed: 00:02:07
   tracing.eventpipe.XUnitWrapper: [Long Running Test] 'tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh', Elapsed: 00:04:07
   tracing.eventpipe.XUnitWrapper: [Long Running Test] 'tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh', Elapsed: 00:06:07
   tracing.eventpipe.XUnitWrapper: [Long Running Test] 'tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh', Elapsed: 00:08:07
Attempting to collect crash dump: /home/helixbot/dotnetbuild/dumps/crashdump_412.dmp
Invoking: sudo /root/helix/work/correlation/createdump --name "/home/helixbot/dotnetbuild/dumps/crashdump_412.dmp" 412 --withheap
createdump stdout:
Gathering state for process 412 corerun

createdump stderr:
CLRDataCreateInstance(ICLRDataEnumMemoryRegions) FAILED 80131c4f

Failed to collect crash dump
    tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh [FAIL]
      
      cmdLine:/root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 9/24/2021 11:48:22 PM, end: 9/24/2021 11:58:22 PM)
      
      Return code:      -100
      Raw output file:      /root/helix/work/workitem/uploads/Reports/tracing.eventpipe/reverseouter/reverseouter/reverseouter.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false reverseouter.dll ''
        0.0s: ::== Running test: TEST_ReverseConnectionCanRecycleWhileTracing
        0.0s: Server name is '/tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4'
        0.1s: running sub-process: /root/helix/work/correlation/corerun /root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll 0
        0.2s: subprocess started: True
        0.2s: subprocess PID: 427
        0.2s: Starting EventPipeSession over standard connection
        0.3s: Waiting for connection
        0.3s: Started EventPipeSession over standard connection with session id: 0x7f9c0033b0
        0.3s: Got a connection
        0.3s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
        0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
        0.4s: Waiting for connection
        0.4s: Got a connection
        0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
        0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
        0.4s: Waiting for connection
        0.4s: Got a connection
        0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
        0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
        0.4s: Waiting for connection
        0.4s: Got a connection
        0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
        0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
        0.4s: Waiting for connection
        0.4s: Got a connection
        0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
        0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
        0.4s: Waiting for connection
        0.8s: Stopping EventPipeSession over standard connection
        2.3s: ================= Subprocess Exited =================
        2.3s: - Exit code: 134
        2.3s: Subprocess stdout: 
        0.3s: =================================================================
        0.3s: Native Crash Reporting
        0.3s: =================================================================
        0.3s: Got a SIGSEGV while executing native code. This usually indicates
        0.3s: a fatal error in the mono runtime or one of the native libraries 
        0.3s: used by your application.
        0.3s: =================================================================
        0.3s: =================================================================
        0.3s: Native stacktrace:
        0.3s: =================================================================
        0.3s: 0x7fae6afd14 - Unknown
        2.1s: =================================================================
        2.1s: Basic Fault Address Reporting
        2.1s: =================================================================
        2.1s: Memory around native instruction pointer (0x7fae6b1acc):0x7fae6b1abc  08 01 40 f9 88 ff ff b4 09 05 40 f9 a9 ff ff b4  ..@.......@.....
        2.1s: 0x7fae6b1acc  2a 09 40 f9 6a ff ff b4 2a 39 40 f9 2a ff ff b4  *[email protected]...*9@.*...
        2.1s: 0x7fae6b1adc  29 01 40 f9 28 1c 00 f9 29 24 01 a9 89 01 00 b4  ).@.(...)$......
        2.1s: 0x7fae6b1aec  2a 05 40 39 aa 00 28 37 2a 05 40 79 6b 00 82 52  *.@9..(7*[email protected]
        2.1s: =================================================================
        2.1s: Managed Stacktrace:
        2.1s: =================================================================
        2.1s: =================================================================
        2.3s: Subprocess stderr: 
        0.3s: =================================================================
        0.3s: External Debugger Dump:
        0.3s: =================================================================
        0.5s: [New LWP 430]
        0.5s: [New LWP 432]
        0.5s: [New LWP 433]
        0.5s: [New LWP 439]
        0.5s: [New LWP 440]
        0.8s: [Thread debugging using libthread_db enabled]
        0.8s: Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
        1.7s: 0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55a68544c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        1.7s: 205../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
        1.7s:   Id   Target Id         Frame 
        1.7s: * 1    Thread 0x7faeba6020 (LWP 427) "corerun" 0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55a68544c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        1.7s:   2    Thread 0x7fadbff1d0 (LWP 430) "SGen worker" 0x0000007faeb512a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fae771630 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        1.7s:   3    Thread 0x7fac00f1d0 (LWP 432) "corerun" 0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
        1.7s:   4    Thread 0x7fabd0e1d0 (LWP 433) "Finalizer" 0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fae762600 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        1.7s:   5    Thread 0x7fab61c1d0 (LWP 439) "corerun" 0x0000007faeb55d5c in __waitpid (pid=<optimized out>, stat_loc=0x7fab6183a0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
        1.7s:   6    Thread 0x7fab41b1d0 (LWP 440) "corerun" 0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
        1.7s: Thread 6 (Thread 0x7fab41b1d0 (LWP 440)):
        1.7s: #0  0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
        1.7s: #1  0x0000007fae57f918 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7fab419700) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
        1.7s: #2  <signal handler called>
        1.7s: #3  0x0000007fae85776c in __clock_nanosleep (clock_id=<optimized out>, flags=-1368247668, req=0x7fab41a920, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:45
        1.8s: #4  0x0000007fae6eb038 in monoeg_g_usleep (microseconds=100000) at /__w/1/s/src/mono/mono/eglib/gdate-unix.c:67
        1.8s: #5  0x0000007fae72328c in ep_rt_thread_sleep (ns=100000000) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1306
        1.8s: #6  streaming_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ep-session.c:68
        1.8s: #7  0x0000007fae723168 in ep_rt_thread_mono_start_func (data=0x7f9c04f920) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1266
        1.8s: #8  0x0000007faeb4b088 in start_thread (arg=0x7fac00e70f) at pthread_create.c:463
        1.8s: #9  0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
        1.8s: Thread 5 (Thread 0x7fab61c1d0 (LWP 439)):
        1.8s: #0  0x0000007faeb55d5c in __waitpid (pid=<optimized out>, stat_loc=0x7fab6183a0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
        1.8s: #1  0x0000007fae6afe14 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:842
        1.8s: #2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7fab618e00, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:869
        1.8s: #3  0x0000007fae66dce0 in mono_handle_native_crash (signal=0x7fae45c2e3 "SIGSEGV", mctx=0x7fab618e00, info=0x7fab619160) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2940
        1.8s: #4  0x0000007fae5dc2d0 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7fab619160, context=0x7fab6191e0, debug_fault_addr=0x357f6a9026008) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3708
        1.8s: #5  <signal handler called>
        1.8s: #6  0x0000007fae6b1acc in interp_frame_iter_next (iter=0x7fab61a428, frame=0x7fab61a670) at /__w/1/s/src/mono/mono/mini/interp/interp.c:7120
        1.9s: #7  0x0000007fae66bc6c in unwinder_unwind_frame (unwinder=0x7fab61a420, jit_tls=<optimized out>, prev_ji=0x0, ctx=<optimized out>, new_ctx=<optimized out>, trace=0x0, lmf=<optimized out>, save_locations=<optimized out>, frame=0x7fab61a670) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:745
        1.9s: #8  0x0000007fae66b74c in mono_walk_stack_full (func=<optimized out>, start_ctx=<optimized out>, jit_tls=<optimized out>, lmf=0x55a68b71e0, unwind_options=<optimized out>, user_data=<optimized out>, crash_context=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:1306
        1.9s: #9  0x0000007fae669c88 in mono_walk_stack_with_state (func=0x7fae724820 <eventpipe_sample_profiler_walk_managed_stack_for_thread_func>, state=<optimized out>, unwind_options=MONO_UNWIND_NONE, user_data=0x7f940032d0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:1204
        1.9s: #10 0x0000007fae724688 in ep_rt_mono_sample_profiler_write_sampling_event_for_threads (sampling_thread=0x7f94000b20, sampling_event=0x7f9c04f5f0) at /__w/1/s/src/mono/mono/eventpipe/ep-rt-mono.c:2550
        1.9s: #11 0x0000007fae7230ac in ep_rt_sample_profiler_write_sampling_event_for_threads (sampling_thread=0x7fab61a428, sampling_event=0x7fab61a670) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:960
        1.9s: #12 sampling_thread (data=0x7f9c04f7a0) at /__w/1/s/src/native/eventpipe/ep-sample-profiler.c:108
        1.9s: #13 0x0000007fae723168 in ep_rt_thread_mono_start_func (data=0x7f9c04f7a0) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1266
        1.9s: #14 0x0000007faeb4b088 in start_thread (arg=0x7fac00e69f) at pthread_create.c:463
        1.9s: #15 0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
        1.9s: Thread 4 (Thread 0x7fabd0e1d0 (LWP 433)):
        1.9s: #0  0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fae762600 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        1.9s: #1  do_futex_wait (sem=sem@entry=0x7fae762600 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
        1.9s: #2  0x0000007faeb53b60 in __new_sem_wait_slow (sem=0x7fae762600 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
        1.9s: #3  0x0000007fae544b10 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
        1.9s: #4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
        1.9s: #5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:876
        1.9s: #6  0x0000007fae51f1e8 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1200
        1.9s: #7  0x0000007fae51f094 in start_wrapper (data=0x55a68de9e0) at /__w/1/s/src/mono/mono/metadata/threads.c:1262
        1.9s: #8  0x0000007faeb4b088 in start_thread (arg=0x7fed437b7f) at pthread_create.c:463
        1.9s: #9  0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
        1.9s: Thread 3 (Thread 0x7fac00f1d0 (LWP 432)):
        1.9s: #0  0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
        1.9s: #1  0x0000007fae57f918 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7fac00d510) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
        1.9s: #2  <signal handler called>
        1.9s: #3  0x0000007fae840e2c in __GI___poll (fds=0x7f9c050340, nfds=548384672446, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
        1.9s: #4  0x0000007fae73cbac in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=8) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
        1.9s: #5  ds_ipc_poll (poll_handles_data=0x7f9c04fab0, poll_handles_data_len=1, timeout_ms=8, callback=0x7fae73c014 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1082
        1.9s: #6  0x0000007fae73a2f0 in ds_ipc_stream_factory_get_next_available_stream (callback=0x7fae73c014 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
        1.9s: #7  0x0000007fae738c7c in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
        1.9s: #8  0x0000007fae73bff4 in ep_rt_thread_mono_start_func (data=0x55a68c2f00) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1266
        1.9s: #9  0x0000007faeb4b088 in start_thread (arg=0x7fed437ccf) at pthread_create.c:463
        1.9s: #10 0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
        1.9s: Thread 2 (Thread 0x7fadbff1d0 (LWP 430)):
        1.9s: #0  0x0000007faeb512a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fae771630 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        1.9s: #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fae7715d8 <lock>, cond=0x7fae771608 <work_cond>) at pthread_cond_wait.c:502
        1.9s: #2  __pthread_cond_wait (cond=0x7fae771608 <work_cond>, mutex=0x7fae7715d8 <lock>) at pthread_cond_wait.c:655
        1.9s: #3  0x0000007fae5c453c in mono_os_cond_wait (cond=0x7fae771630 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
        1.9s: #4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
        1.9s: #5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
        1.9s: #6  0x0000007faeb4b088 in start_thread (arg=0x7fed437c6f) at pthread_create.c:463
        1.9s: #7  0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
        1.9s: Thread 1 (Thread 0x7faeba6020 (LWP 427)):
        1.9s: #0  0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55a68544c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        1.9s: #1  do_futex_wait (sem=sem@entry=0x55a68544c0, abstime=0x0) at sem_waitcommon.c:111
        2.0s: #2  0x0000007faeb53b60 in __new_sem_wait_slow (sem=0x55a68544c0, abstime=0x0) at sem_waitcommon.c:181
        2.0s: #3  0x0000007fae579c6c in mono_os_sem_wait (sem=0x55a68544c0, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
        2.0s: #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
        2.0s: #5  0x0000007fae5803ec in mono_threads_enter_gc_unsafe_region_unbalanced_with_info (info=0x55a6854460, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:496
        2.0s: #6  0x0000007fae6be158 in ves_pinvoke_method (imethod=0x55a6b8a768, sig=0x55a6b9f500, addr=0x7fae5cb9c0 <GlobalizationNative_GetLocaleInfoString>, context=0x55a68c3a40, parent_frame=0x7fed4376a0, ret_sp=0x7fabd0fb40, sp=0x7fabd0fb48, save_last_error=0, cache=0x55a6b8a960) at /__w/1/s/src/mono/mono/mini/interp/interp.c:1582
        2.0s: #7  0x0000007fae6b3050 in interp_exec_method (frame=<optimized out>, context=<optimized out>, clause_args=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:3553
        2.0s: #8  0x0000007fae6b1488 in interp_runtime_invoke (method=<optimized out>, obj=0x0, params=0x7fed437fa8, exc=0x0, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:1948
        2.0s: #9  0x0000007fae5060a4 in do_runtime_invoke (method=0x55a683c578, obj=0x0, params=0x7fed437fa8, exc=0x0, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:2393
        2.0s: #10 mono_runtime_invoke_checked (method=0x55a683c578, obj=0x0, params=0x7fed437fa8, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:2561
        2.0s: #11 0x0000007fae50cc90 in do_exec_main_checked (method=0x55a683c578, args=<optimized out>, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:4397
        2.0s: #12 mono_runtime_exec_main_checked (method=0x55a683c578, args=<optimized out>, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:4503
        2.0s: #13 0x0000007fae630e0c in mono_jit_exec_internal (domain=<optimized out>, assembly=<optimized out>, argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:1379
        2.0s: #14 mono_jit_exec (domain=<optimized out>, assembly=<optimized out>, argc=2, argv=0x55a684afd0) at /__w/1/s/src/mono/mono/mini/driver.c:1324
        2.0s: #15 0x0000007fae633a58 in main_thread_handler (user_data=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:1456
        2.0s: #16 mono_main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:2677
        2.0s: #17 0x0000007fae69d0a4 in monovm_execute_assembly (argc=1, argv=0x55a6842f60, managedAssemblyPath=<optimized out>, exitCode=0x7fed438304) at /__w/1/s/src/mono/mono/mini/monovm.c:267
        2.0s: #18 0x000000556b0580fc in run (config=...) at /__w/1/s/src/coreclr/hosts/corerun/corerun.cpp:368
        2.0s: #19 main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/coreclr/hosts/corerun/corerun.cpp:563
        2.3s: System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
       ---> System.Net.Sockets.SocketException (104): Connection reset by peer
         at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 1446
         at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 231
         --- End of inner exception stack trace ---
         at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 235
         at System.IO.BinaryReader.ReadBytes(Int32 count) in /_/src/libraries/System.Private.CoreLib/src/System/IO/BinaryReader.cs:line 498
         at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcHeader.TryParse(BinaryReader reader)
         at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcMessage.Parse(Stream stream)
         at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.Read(Stream stream)
         at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.SendMessage(Int32 processId, IpcMessage message)
         at Microsoft.Diagnostics.Tools.RuntimeClient.EventPipeClient.StopTracing(Int32 processId, UInt64 sessionId)
         at Tracing.Tests.ReverseValidation.ReverseValidation.<>c__DisplayClass0_1.<<TEST_ReverseConnectionCanRecycleWhileTracing>b__1>d.MoveNext()
        2.3s: setting the MRE
      
      cmdLine:/root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 9/24/2021 11:48:22 PM, end: 9/24/2021 11:58:22 PM)
      Test Harness Exitcode is : -100
      To run the test:
      > set CORE_ROOT=/root/helix/work/correlation
      > /root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at tracing_eventpipe._reverseouter_reverseouter_reverseouter_._reverseouter_reverseouter_reverseouter_sh()
      Output:
        
        cmdLine:/root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 9/24/2021 11:48:22 PM, end: 9/24/2021 11:58:22 PM)
        
        Return code:      -100
        Raw output file:      /root/helix/work/workitem/uploads/Reports/tracing.eventpipe/reverseouter/reverseouter/reverseouter.output.txt
        Raw output:
        BEGIN EXECUTION
        /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false reverseouter.dll ''
          0.0s: ::== Running test: TEST_ReverseConnectionCanRecycleWhileTracing
          0.0s: Server name is '/tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4'
          0.1s: running sub-process: /root/helix/work/correlation/corerun /root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.dll 0
          0.2s: subprocess started: True
          0.2s: subprocess PID: 427
          0.2s: Starting EventPipeSession over standard connection
          0.3s: Waiting for connection
          0.3s: Started EventPipeSession over standard connection with session id: 0x7f9c0033b0
          0.3s: Got a connection
          0.3s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
          0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
          0.4s: Waiting for connection
          0.4s: Got a connection
          0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
          0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
          0.4s: Waiting for connection
          0.4s: Got a connection
          0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
          0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
          0.4s: Waiting for connection
          0.4s: Got a connection
          0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
          0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
          0.4s: Waiting for connection
          0.4s: Got a connection
          0.4s: Shutting down Reverse Server at /tmp/DOTNET_TRACE_TESTS_hyloxa5a.po4
          0.4s: { Magic=System.Byte[]; ClrInstanceId=e6267599-d374-4ff7-8ce7-e4b5cda4bead; ProcessId=427; Unused=0; }
          0.4s: Waiting for connection
          0.8s: Stopping EventPipeSession over standard connection
          2.3s: ================= Subprocess Exited =================
          2.3s: - Exit code: 134
          2.3s: Subprocess stdout: 
          0.3s: =================================================================
          0.3s: Native Crash Reporting
          0.3s: =================================================================
          0.3s: Got a SIGSEGV while executing native code. This usually indicates
          0.3s: a fatal error in the mono runtime or one of the native libraries 
          0.3s: used by your application.
          0.3s: =================================================================
          0.3s: =================================================================
          0.3s: Native stacktrace:
          0.3s: =================================================================
          0.3s: 0x7fae6afd14 - Unknown
          2.1s: =================================================================
          2.1s: Basic Fault Address Reporting
          2.1s: =================================================================
          2.1s: Memory around native instruction pointer (0x7fae6b1acc):0x7fae6b1abc  08 01 40 f9 88 ff ff b4 09 05 40 f9 a9 ff ff b4  ..@.......@.....
          2.1s: 0x7fae6b1acc  2a 09 40 f9 6a ff ff b4 2a 39 40 f9 2a ff ff b4  *[email protected]...*9@.*...
          2.1s: 0x7fae6b1adc  29 01 40 f9 28 1c 00 f9 29 24 01 a9 89 01 00 b4  ).@.(...)$......
          2.1s: 0x7fae6b1aec  2a 05 40 39 aa 00 28 37 2a 05 40 79 6b 00 82 52  *.@9..(7*[email protected]
          2.1s: =================================================================
          2.1s: Managed Stacktrace:
          2.1s: =================================================================
          2.1s: =================================================================
          2.3s: Subprocess stderr: 
          0.3s: =================================================================
          0.3s: External Debugger Dump:
          0.3s: =================================================================
          0.5s: [New LWP 430]
          0.5s: [New LWP 432]
          0.5s: [New LWP 433]
          0.5s: [New LWP 439]
          0.5s: [New LWP 440]
          0.8s: [Thread debugging using libthread_db enabled]
          0.8s: Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
          1.7s: 0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55a68544c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
          1.7s: 205../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
          1.7s:   Id   Target Id         Frame 
          1.7s: * 1    Thread 0x7faeba6020 (LWP 427) "corerun" 0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55a68544c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
          1.7s:   2    Thread 0x7fadbff1d0 (LWP 430) "SGen worker" 0x0000007faeb512a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fae771630 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
          1.7s:   3    Thread 0x7fac00f1d0 (LWP 432) "corerun" 0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
          1.7s:   4    Thread 0x7fabd0e1d0 (LWP 433) "Finalizer" 0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fae762600 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
          1.7s:   5    Thread 0x7fab61c1d0 (LWP 439) "corerun" 0x0000007faeb55d5c in __waitpid (pid=<optimized out>, stat_loc=0x7fab6183a0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
          1.7s:   6    Thread 0x7fab41b1d0 (LWP 440) "corerun" 0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
          1.7s: Thread 6 (Thread 0x7fab41b1d0 (LWP 440)):
          1.7s: #0  0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
          1.7s: #1  0x0000007fae57f918 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7fab419700) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
          1.7s: #2  <signal handler called>
          1.7s: #3  0x0000007fae85776c in __clock_nanosleep (clock_id=<optimized out>, flags=-1368247668, req=0x7fab41a920, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:45
          1.8s: #4  0x0000007fae6eb038 in monoeg_g_usleep (microseconds=100000) at /__w/1/s/src/mono/mono/eglib/gdate-unix.c:67
          1.8s: #5  0x0000007fae72328c in ep_rt_thread_sleep (ns=100000000) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1306
          1.8s: #6  streaming_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ep-session.c:68
          1.8s: #7  0x0000007fae723168 in ep_rt_thread_mono_start_func (data=0x7f9c04f920) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1266
          1.8s: #8  0x0000007faeb4b088 in start_thread (arg=0x7fac00e70f) at pthread_create.c:463
          1.8s: #9  0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
          1.8s: Thread 5 (Thread 0x7fab61c1d0 (LWP 439)):
          1.8s: #0  0x0000007faeb55d5c in __waitpid (pid=<optimized out>, stat_loc=0x7fab6183a0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
          1.8s: #1  0x0000007fae6afe14 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:842
          1.8s: #2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0x7fab618e00, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:869
          1.8s: #3  0x0000007fae66dce0 in mono_handle_native_crash (signal=0x7fae45c2e3 "SIGSEGV", mctx=0x7fab618e00, info=0x7fab619160) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2940
          1.8s: #4  0x0000007fae5dc2d0 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7fab619160, context=0x7fab6191e0, debug_fault_addr=0x357f6a9026008) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3708
          1.8s: #5  <signal handler called>
          1.8s: #6  0x0000007fae6b1acc in interp_frame_iter_next (iter=0x7fab61a428, frame=0x7fab61a670) at /__w/1/s/src/mono/mono/mini/interp/interp.c:7120
          1.9s: #7  0x0000007fae66bc6c in unwinder_unwind_frame (unwinder=0x7fab61a420, jit_tls=<optimized out>, prev_ji=0x0, ctx=<optimized out>, new_ctx=<optimized out>, trace=0x0, lmf=<optimized out>, save_locations=<optimized out>, frame=0x7fab61a670) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:745
          1.9s: #8  0x0000007fae66b74c in mono_walk_stack_full (func=<optimized out>, start_ctx=<optimized out>, jit_tls=<optimized out>, lmf=0x55a68b71e0, unwind_options=<optimized out>, user_data=<optimized out>, crash_context=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:1306
          1.9s: #9  0x0000007fae669c88 in mono_walk_stack_with_state (func=0x7fae724820 <eventpipe_sample_profiler_walk_managed_stack_for_thread_func>, state=<optimized out>, unwind_options=MONO_UNWIND_NONE, user_data=0x7f940032d0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:1204
          1.9s: #10 0x0000007fae724688 in ep_rt_mono_sample_profiler_write_sampling_event_for_threads (sampling_thread=0x7f94000b20, sampling_event=0x7f9c04f5f0) at /__w/1/s/src/mono/mono/eventpipe/ep-rt-mono.c:2550
          1.9s: #11 0x0000007fae7230ac in ep_rt_sample_profiler_write_sampling_event_for_threads (sampling_thread=0x7fab61a428, sampling_event=0x7fab61a670) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:960
          1.9s: #12 sampling_thread (data=0x7f9c04f7a0) at /__w/1/s/src/native/eventpipe/ep-sample-profiler.c:108
          1.9s: #13 0x0000007fae723168 in ep_rt_thread_mono_start_func (data=0x7f9c04f7a0) at /__w/1/s/src/mono/mono/mini/../../mono/eventpipe/ep-rt-mono.h:1266
          1.9s: #14 0x0000007faeb4b088 in start_thread (arg=0x7fac00e69f) at pthread_create.c:463
          1.9s: #15 0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
          1.9s: Thread 4 (Thread 0x7fabd0e1d0 (LWP 433)):
          1.9s: #0  0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fae762600 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
          1.9s: #1  do_futex_wait (sem=sem@entry=0x7fae762600 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
          1.9s: #2  0x0000007faeb53b60 in __new_sem_wait_slow (sem=0x7fae762600 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
          1.9s: #3  0x0000007fae544b10 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
          1.9s: #4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
          1.9s: #5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:876
          1.9s: #6  0x0000007fae51f1e8 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1200
          1.9s: #7  0x0000007fae51f094 in start_wrapper (data=0x55a68de9e0) at /__w/1/s/src/mono/mono/metadata/threads.c:1262
          1.9s: #8  0x0000007faeb4b088 in start_thread (arg=0x7fed437b7f) at pthread_create.c:463
          1.9s: #9  0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
          1.9s: Thread 3 (Thread 0x7fac00f1d0 (LWP 432)):
          1.9s: #0  0x0000007fae7ac88c in __GI___sigsuspend (set=0x7fae767828 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
          1.9s: #1  0x0000007fae57f918 in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7fac00d510) at /__w/1/s/src/mono/mono/utils/mono-threads-posix-signals.c:200
          1.9s: #2  <signal handler called>
          1.9s: #3  0x0000007fae840e2c in __GI___poll (fds=0x7f9c050340, nfds=548384672446, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
          1.9s: #4  0x0000007fae73cbac in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=8) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
          1.9s: #5  ds_ipc_poll (poll_handles_data=0x7f9c04fab0, poll_handles_data_len=1, timeout_ms=8, callback=0x7fae73c014 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1082
          1.9s: #6  0x0000007fae73a2f0 in ds_ipc_stream_factory_get_next_available_stream (callback=0x7fae73c014 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
          1.9s: #7  0x0000007fae738c7c in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
          1.9s: #8  0x0000007fae73bff4 in ep_rt_thread_mono_start_func (data=0x55a68c2f00) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1266
          1.9s: #9  0x0000007faeb4b088 in start_thread (arg=0x7fed437ccf) at pthread_create.c:463
          1.9s: #10 0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
          1.9s: Thread 2 (Thread 0x7fadbff1d0 (LWP 430)):
          1.9s: #0  0x0000007faeb512a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fae771630 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
          1.9s: #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fae7715d8 <lock>, cond=0x7fae771608 <work_cond>) at pthread_cond_wait.c:502
          1.9s: #2  __pthread_cond_wait (cond=0x7fae771608 <work_cond>, mutex=0x7fae7715d8 <lock>) at pthread_cond_wait.c:655
          1.9s: #3  0x0000007fae5c453c in mono_os_cond_wait (cond=0x7fae771630 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
          1.9s: #4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
          1.9s: #5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
          1.9s: #6  0x0000007faeb4b088 in start_thread (arg=0x7fed437c6f) at pthread_create.c:463
          1.9s: #7  0x0000007fae849ffc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
          1.9s: Thread 1 (Thread 0x7faeba6020 (LWP 427)):
          1.9s: #0  0x0000007faeb53a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55a68544c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
          1.9s: #1  do_futex_wait (sem=sem@entry=0x55a68544c0, abstime=0x0) at sem_waitcommon.c:111
          2.0s: #2  0x0000007faeb53b60 in __new_sem_wait_slow (sem=0x55a68544c0, abstime=0x0) at sem_waitcommon.c:181
          2.0s: #3  0x0000007fae579c6c in mono_os_sem_wait (sem=0x55a68544c0, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
          2.0s: #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
          2.0s: #5  0x0000007fae5803ec in mono_threads_enter_gc_unsafe_region_unbalanced_with_info (info=0x55a6854460, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:496
          2.0s: #6  0x0000007fae6be158 in ves_pinvoke_method (imethod=0x55a6b8a768, sig=0x55a6b9f500, addr=0x7fae5cb9c0 <GlobalizationNative_GetLocaleInfoString>, context=0x55a68c3a40, parent_frame=0x7fed4376a0, ret_sp=0x7fabd0fb40, sp=0x7fabd0fb48, save_last_error=0, cache=0x55a6b8a960) at /__w/1/s/src/mono/mono/mini/interp/interp.c:1582
          2.0s: #7  0x0000007fae6b3050 in interp_exec_method (frame=<optimized out>, context=<optimized out>, clause_args=0x0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:3553
          2.0s: #8  0x0000007fae6b1488 in interp_runtime_invoke (method=<optimized out>, obj=0x0, params=0x7fed437fa8, exc=0x0, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/mini/interp/interp.c:1948
          2.0s: #9  0x0000007fae5060a4 in do_runtime_invoke (method=0x55a683c578, obj=0x0, params=0x7fed437fa8, exc=0x0, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:2393
          2.0s: #10 mono_runtime_invoke_checked (method=0x55a683c578, obj=0x0, params=0x7fed437fa8, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:2561
          2.0s: #11 0x0000007fae50cc90 in do_exec_main_checked (method=0x55a683c578, args=<optimized out>, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:4397
          2.0s: #12 mono_runtime_exec_main_checked (method=0x55a683c578, args=<optimized out>, error=0x7fed437ff0) at /__w/1/s/src/mono/mono/metadata/object.c:4503
          2.0s: #13 0x0000007fae630e0c in mono_jit_exec_internal (domain=<optimized out>, assembly=<optimized out>, argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:1379
          2.0s: #14 mono_jit_exec (domain=<optimized out>, assembly=<optimized out>, argc=2, argv=0x55a684afd0) at /__w/1/s/src/mono/mono/mini/driver.c:1324
          2.0s: #15 0x0000007fae633a58 in main_thread_handler (user_data=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:1456
          2.0s: #16 mono_main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:2677
          2.0s: #17 0x0000007fae69d0a4 in monovm_execute_assembly (argc=1, argv=0x55a6842f60, managedAssemblyPath=<optimized out>, exitCode=0x7fed438304) at /__w/1/s/src/mono/mono/mini/monovm.c:267
          2.0s: #18 0x000000556b0580fc in run (config=...) at /__w/1/s/src/coreclr/hosts/corerun/corerun.cpp:368
          2.0s: #19 main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/coreclr/hosts/corerun/corerun.cpp:563
          2.3s: System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
         ---> System.Net.Sockets.SocketException (104): Connection reset by peer
           at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs:line 1446
           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 231
           --- End of inner exception stack trace ---
           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/NetworkStream.cs:line 235
           at System.IO.BinaryReader.ReadBytes(Int32 count) in /_/src/libraries/System.Private.CoreLib/src/System/IO/BinaryReader.cs:line 498
           at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcHeader.TryParse(BinaryReader reader)
           at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcMessage.Parse(Stream stream)
           at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.Read(Stream stream)
           at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.SendMessage(Int32 processId, IpcMessage message)
           at Microsoft.Diagnostics.Tools.RuntimeClient.EventPipeClient.StopTracing(Int32 processId, UInt64 sessionId)
           at Tracing.Tests.ReverseValidation.ReverseValidation.<>c__DisplayClass0_1.<<TEST_ReverseConnectionCanRecycleWhileTracing>b__1>d.MoveNext()
          2.3s: setting the MRE
        
        cmdLine:/root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 9/24/2021 11:48:22 PM, end: 9/24/2021 11:58:22 PM)
        Test Harness Exitcode is : -100
        To run the test:
        > set CORE_ROOT=/root/helix/work/correlation
        > /root/helix/work/workitem/e/tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh
  Finished:    tracing.eventpipe.XUnitWrapper
  Discovering: tracing.eventcounter.XUnitWrapper (method display = ClassAndMethod, method display options = None)
@dotnet-issue-labeler dotnet-issue-labeler bot added area-Tracing-coreclr untriaged New issue has not been triaged by the area owner labels Sep 29, 2021
@tommcdon
Copy link
Member

tommcdon commented Oct 3, 2021

@lateralusX @josalem

@lateralusX
Copy link
Member

lateralusX commented Oct 4, 2021 •
edited
Loading

Very likely caused by this issue: #59054

Looking that the crash we have a thread that is suspended when doing a pinvoke:

mono_threads_enter_gc_unsafe_region_unbalanced_with_info
ves_pinvoke_method
interp_exec_method

and then we have a crash unwinding the stack from sample profiler, so all condition as described in that issue applies to this crash.

@josalem
Copy link
Contributor

josalem commented Oct 4, 2021

@lateralusX are you okay closing this as a dup of #59054?

@josalem josalem removed the untriaged New issue has not been triaged by the area owner label Oct 4, 2021
@lateralusX
Copy link
Member

Dup of #59054.

@ghost ghost locked as resolved and limited conversation to collaborators Nov 4, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Tracing-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@danmoseley @lateralusX @tommcdon @josalem