Long running test: InvalidJsonForTypeShouldFail

[ViktorHofer]

"C:\h\w\A4A5086D\p\dotnet.exe" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\A4A5086D\w\B746096E\e>"C:\h\w\A4A5086D\p\dotnet.exe" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2106 of 2136 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 2)
   System.Text.Json.Tests: [Long Running Test] 'System.Text.Json.Serialization.Tests.ReferenceHandlerTests.TestJsonPathDoesNotFailOnMultiThreads', Elapsed: 00:05:43
[Long Running Test] 'System.Text.Json.Serialization.Tests.ConstructorTests_String.ReadSimpleObjectAsync', Elapsed: 00:05:08
[Long Running Test] 'System.Text.Json.Serialization.Tests.CacheTests.MultipleTypes', Elapsed: 00:05:41
[Long Running Test] 'System.Text.Json.Tests.Utf8JsonWriterTests.Writing3MBBase64Bytes', Elapsed: 00:02:08
[Long Running Test] 'System.Text.Json.Serialization.Tests.StreamTests.RoundTripAsync', Elapsed: 00:05:06
[Long Running Test] 'System.Text.Json.Serialization.Tests.InvalidJsonTests.InvalidJsonForTypeShouldFail', Elapsed: 00:04:58

https://dev.azure.com/dnceng/public/_build/results?buildId=833169&view=ms.vss-test-web.build-test-results-tab

Configuration: net5.0-Windows_NT-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open

@layomia @steveharter @jozkee please take a look. Seems to happening in multiple builds.

[ericstj]

I give this line an A+ for foreshadowing:

runtime/src/libraries/System.Text.Json/tests/Serialization/InvalidJsonTests.cs

Line 321 in 7cd7783

foreach (object[] args in DataForInvalidJsonForTypeTests()) // ~140K tests, too many for theory to handle well with our infrastructure

@layomia what's supposed to be happening here?

[layomia]

@ericstj the test is passing many variations of invalid JSON to the deserializer (mapping to complex objects like lists, dictionaries, and objects) to make sure that we always throw a sane exception rather than having bad bugs like leaking NullReferenceException and InvalidOperationException. This was in response to some bugs that were discovered late in the 3.1 release - #31309, #31192.

We could investigate seeing if we can reduce the number of test cases here (from ~140k) to something more sane, but I think making the test outer-loop would work well since we don't expect many inner-loop changes at this stage that could make them fail.

[ViktorHofer]

From what I see we run those 140k test rows in serial. Can we add some parallelization to make this faster? ie Parallel.ForEach

[ViktorHofer]

@layomia we need to do something about this (see #42677 for all the data). I tend towards disabling the test by EOD. Opinions?

[layomia]

@ViktorHofer - I opened a PR #42960.