Log ulimit and related values for test failures #46511

danmoseley · 2021-01-03T00:08:08Z

I want to understand why we are not getting dumps for cases like this

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-e8d9bc08dc2441d7a0/System.IO.FileSystem.Tests/console.547b5a51.log?sv=2019-07-07&se=2021-01-06T01%3A52%3A38Z&sr=c&sp=rl&sig=DG1seb%2FgKbIzrNgJNwQwHSbwyf0EG5DTpgG8jpxfwZk%3D

   at System.Threading.ThreadHelper.ThreadStart(System.Object)
./RunTests.sh: line 161:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Thu Dec 17 02:03:57 UTC 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
Waiting a few seconds for any dump to be written..
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem
+ export '_commandExitCode=134'

Although we are currently setting ulimit in this file, if I understand correctly part or all of the responsibility is on Helix (eg this) so the work done in here to set these may be redundant. Still it seems like a useful place to at least log the values.

ghost · 2021-01-03T00:08:12Z

Tagging subscribers to this area: @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

I want to understand why we are not getting dumps for cases like this

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-e8d9bc08dc2441d7a0/System.IO.FileSystem.Tests/console.547b5a51.log?sv=2019-07-07&se=2021-01-06T01%3A52%3A38Z&sr=c&sp=rl&sig=DG1seb%2FgKbIzrNgJNwQwHSbwyf0EG5DTpgG8jpxfwZk%3D

   at System.Threading.ThreadHelper.ThreadStart(System.Object)
./RunTests.sh: line 161:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Thu Dec 17 02:03:57 UTC 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
Waiting a few seconds for any dump to be written..
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem
+ export '_commandExitCode=134'

Although we are currently setting ulimit in this file, if I understand correctly part or all of the responsibility is on Helix (eg this) so the work done in here to set these may be redundant. Still it seems like a useful place to at least log the values.

Author:	danmosemsft
Assignees:	-
Labels:	`area-Infrastructure-libraries`
Milestone:	-

danmoseley · 2021-01-03T00:08:28Z

relates to #46381

danmoseley · 2021-01-03T00:25:12Z

Another more recent example taken at random
Console log: 'System.Text.Json.Tests' from job b6a221f0-771f-4f96-ba08-76f55a599f26 (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855 on ddvsotx2l133

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b6a221f0771f4f96ba/System.Text.Json.Tests/console.d9b451ca.log?sv=2019-07-07&se=2021-01-15T09%3A48%3A52Z&sr=c&sp=rl&sig=DrPOgRxgnrCcxNsL923nsqWqnyGx4aoO0KPPQ898Rrw%3D

Assert failure(PID 133 [0x00000085], Thread: 146 [0x0092]): Assertion failed 'isValidGeneralDatasize(size)' in 'Node[Nullable`1][System.Nullable`1[System.Int64]]:NodeTreeFromList(System.Collections.Immutable.IOrderedCollection`1[Nullable`1],int,int):Node[Nullable`1]' during 'Generate code' (IL size 108)

    File: /__w/1/s/src/coreclr/jit/emitarm64.cpp Line: 7689
    Image: /root/helix/work/correlation/dotnet

./RunTests.sh: line 161:   133 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Sat Dec 26 10:04:31 UTC 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
Waiting a few seconds for any dump to be written..
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem
+ export _commandExitCode=134

danmoseley · 2021-01-03T04:05:44Z

sample output on Linux

  === TEST EXECUTION SUMMARY ===
     System.Runtime.InteropServices.RuntimeInformation.Tests  Total: 13, Errors: 0, Failed: 1, Skipped: 0, Time: 0.554s
  ~/git/runtime/src/libraries/System.Runtime.InteropServices.RuntimeInformation/tests
  ----- end Sat 02 Jan 2021 03:47:42 PM PST ----- exit code 1 ----------------------------------------------------------
  ulimit -c value: unlimited
  cat /proc/sys/kernel/core_pattern: core
  cat /proc/sys/kernel/core_uses_pid: 0
  cat: /proc/sys/kernel/coredump_filter: No such file or directory
  cat /proc/sys/kernel/coredump_filter:
  Looking around for any Linux dump..
  ... found no dump in /home/dan/git/runtime/artifacts/bin/System.Runtime.InteropServices.RuntimeInformation.Tests/net6.0-Unix-Debug

I don't have a Mac to test.

danmoseley · 2021-01-04T15:30:07Z

I need to test the Mac line.. done

ghost · 2021-01-04T18:36:40Z

Hello @danmosemsft!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

safern · 2021-01-04T21:32:07Z

eng/testing/RunnerTemplate.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env bash


It seems like you're changing the file permissions, is that intentional?

Merged with other PR

log ulimit

2ad0db8

Dotnet-GitSync-Bot added the area-Infrastructure-libraries label Jan 3, 2021

danmoseley requested review from ViktorHofer and safern January 3, 2021 00:08

stephentoub approved these changes Jan 4, 2021

View reviewed changes

danmoseley force-pushed the ulimit.log branch from 48a41b3 to 2ad0db8 Compare January 4, 2021 17:55

runfoapp bot mentioned this pull request Jan 4, 2021

Inability to unzip assets during build on Unix x64 #32805

Closed

danmoseley added the auto-merge label Jan 4, 2021

safern reviewed Jan 4, 2021

View reviewed changes

danmoseley merged commit f640d32 into dotnet:master Jan 4, 2021

ghost locked as resolved and limited conversation to collaborators Feb 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log ulimit and related values for test failures #46511

Log ulimit and related values for test failures #46511

danmoseley commented Jan 3, 2021

ghost commented Jan 3, 2021

danmoseley commented Jan 3, 2021

danmoseley commented Jan 3, 2021

danmoseley commented Jan 3, 2021

danmoseley commented Jan 4, 2021 •

edited

Loading

ghost commented Jan 4, 2021

safern Jan 4, 2021

danmoseley Jan 4, 2021

Log ulimit and related values for test failures #46511

Log ulimit and related values for test failures #46511

Conversation

danmoseley commented Jan 3, 2021

ghost commented Jan 3, 2021

danmoseley commented Jan 3, 2021

danmoseley commented Jan 3, 2021

danmoseley commented Jan 3, 2021

danmoseley commented Jan 4, 2021 • edited Loading

ghost commented Jan 4, 2021

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

safern Jan 4, 2021

Choose a reason for hiding this comment

danmoseley Jan 4, 2021

Choose a reason for hiding this comment

danmoseley commented Jan 4, 2021 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.