Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Net.* cancellation tests fail in CI #72818

Closed
runfoapp bot opened this issue Jul 25, 2022 · 16 comments
Closed

System.Net.* cancellation tests fail in CI #72818

runfoapp bot opened this issue Jul 25, 2022 · 16 comments
Labels
area-System.Net blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone

Comments

@runfoapp
Copy link

runfoapp bot commented Jul 25, 2022

System.Net.Mail.Functional.Tests test timed out in the rolling build 1901005
System.Net.Http.Functional.Tests test timed out with a very similar looking error in rolling build1902723

Between the two tests Runfo reports ~140 failures in the last 7 days. There are also some System.Net.Mail.Functional.Tests that failed in different ways and I am trying to filter those out. Every example from System.Net.Http.Functional.Tests has looked similar so far.

System.Net.Mail.Functional.Tests
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-6bd0a6280a374277a1/System.Net.Mail.Functional.Tests/1/console.531221cb.log?%3Fhelixlogtype%3Dresult

   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.SendMailAsync_CanBeCanceled_CancellationToken', Elapsed: 00:02:08
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.SendMailAsync_CanBeCanceled_CancellationToken', Elapsed: 00:04:08
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.SendMailAsync_CanBeCanceled_CancellationToken', Elapsed: 00:06:08
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.SendMailAsync_CanBeCanceled_CancellationToken', Elapsed: 00:08:08
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.SendMailAsync_CanBeCanceled_CancellationToken', Elapsed: 00:10:08
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.SendMailAsync_CanBeCanceled_CancellationToken', Elapsed: 00:12:08
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.SendMailAsync_CanBeCanceled_CancellationToken', Elapsed: 00:14:08

...
[EXECUTION TIMED OUT]

System.Net.Http.Functional.Tests
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-287f10d639984af991/System.Net.Http.Functional.Tests/1/console.8747f084.log?%3Fhelixlogtype%3Dresult

   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:02:17
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:04:18
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:06:19
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:08:20
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:10:21
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:12:22
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:14:23
['System.Net.Http.Functional.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

Runfo Tracking Issue: system.net.http.functional.tests work item

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1923565 runtime PR 73274 net7.0-windows-Debug-x64-CoreCLR_release-Windows.11.Amd64.Client.Open console.log core dump runclient.py
1921281 runtime PR 73168 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1918362 runtime PR 71180 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1916354 runtime PR 73185 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1916354 runtime PR 73185 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1915051 runtime PR 49576 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1914778 runtime PR 49576 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-windows-Debug-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1913943 runtime PR 49576 net7.0-windows-Debug-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1913943 runtime PR 49576 net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1911388 runtime PR 72851 net7.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504 console.log runclient.py
1911382 runtime PR 73055 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1911382 runtime PR 73055 net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1910251 runtime Rolling net7.0-OSX-Release-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1909465 runtime PR 72979 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log runclient.py
1908206 runtime PR 72979 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log runclient.py
1906782 runtime PR 70582 net7.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504 console.log runclient.py
1906762 runtime PR 72925 net7.0-Linux-Debug-x64-CoreCLR_release-(Centos.7.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af console.log runclient.py
1906736 runtime PR 72940 net7.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open console.log runclient.py
1906658 runtime PR 72937 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1906658 runtime PR 72937 net7.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1906602 runtime PR 70015 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1906413 runtime PR 72932 net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1906378 runtime PR 72933 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1906306 runtime PR 72931 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1906306 runtime PR 72931 net7.0-windows-Release-x86-CoreCLR_release-Windows.7.Amd64.Open console.log runclient.py
1906180 runtime Rolling net7.0-windows-Release-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1906149 runtime PR 72669 net7.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504 console.log runclient.py
1906108 runtime PR 72917 net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1905962 runtime PR 72412 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1905947 runtime PR 72832 net7.0-windows-Debug-x64-CoreCLR_release-Windows.11.Amd64.Client.Open console.log runclient.py
1905835 runtime PR 72434 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1905497 runtime PR 72896 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1905366 runtime PR 72448 net7.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1905224 runtime PR 72869 net7.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1905149 runtime PR 72394 net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log runclient.py
1904930 runtime Rolling net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open console.log runclient.py
1904358 runtime PR 72639 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1904205 runtime PR 71785 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1904182 runtime PR 72863 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1903814 runtime PR 72853 net7.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504 console.log runclient.py
1903704 runtime PR 72847 net7.0-windows-Debug-x64-CoreCLR_release-Windows.11.Amd64.Client.Open console.log runclient.py
1903661 runtime PR 72789 net7.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1903421 runtime PR 72275 net7.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open console.log runclient.py
1903303 runtime PR 71478 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1903279 runtime PR 72832 net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1903279 runtime PR 72832 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1902723 runtime Rolling net7.0-OSX-Release-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1902669 runtime PR 72817 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1902530 runtime PR 72811 net7.0-Linux-Debug-x64-CoreCLR_release-RedHat.7.Amd64.Open console.log runclient.py
1902291 runtime Rolling net7.0-OSX-Release-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1902291 runtime Rolling net7.0-windows-Release-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1901310 runtime PR 72448 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1901165 runtime PR 72779 net7.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1901090 runtime PR 72769 net7.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open console.log runclient.py
1900750 runtime PR 72741 net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1900494 runtime PR 72538 net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log runclient.py
1900382 runtime PR 71308 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1900376 runtime PR 72741 net7.0-windows-Release-x86-CoreCLR_release-Windows.7.Amd64.Open console.log runclient.py
1900369 runtime PR 67198 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1900266 runtime PR 72448 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1900112 runtime Rolling net7.0-OSX-Release-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1899982 runtime PR 72727 net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open console.log runclient.py
1899824 runtime PR 72448 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1899397 runtime PR 72672 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1899390 runtime PR 67861 net7.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server2022.ES.Open console.log runclient.py
1898882 runtime PR 72706 net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1887984 runtime PR 62863 net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1887984 runtime PR 62863 net7.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504 console.log runclient.py
1887984 runtime PR 62863 net7.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)[email protected]/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504 console.log runclient.py
1887984 runtime PR 62863 net7.0-windows-Release-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1841231 runtime PR 71180 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1841231 runtime PR 71180 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1841231 runtime PR 71180 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1841231 runtime PR 71180 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 9 60
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jul 25, 2022
@ghost
Copy link

ghost commented Jul 25, 2022

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo Creating Tracking Issue (data being generated)

Author: runfoapp[bot]
Assignees: -
Labels:

area-System.Net

Milestone: -

@noahfalk noahfalk added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Jul 25, 2022
@danmoseley
Copy link
Member

danmoseley commented Jul 26, 2022

WASM ones are some install failure.

The Mac and Debian failures are assertion failures

dotnet: /__w/1/s/src/native/libs/Common/pal_utilities.h:86: int ToFileDescriptor(intptr_t): Assertion `0 <= fd && fd < sysconf(_SC_OPEN_MAX)' failed.

top of stack looks like

#10 0x00007f4c7b4ab5cd in ToFileDescriptor (fd=-1) at /__w/1/s/src/native/libs/Common/pal_utilities.h:86
#11 0x00007f4c7b4abf95 in SystemNative_FcntlGetFD (fd=-1) at /__w/1/s/src/native/libs/System.Native/pal_io.c:611
...
	  at <unknown> <0xffffffff>
	  at Fcntl:<GetFD>g____PInvoke|5_0 <0x00020>
	  at Fcntl:GetFD <0x00020>
	  at System.Net.Sockets.SafeSocketHandle:TryUnblockSocket <0x0003a>
	  at System.Net.Sockets.SafeSocketHandle:CloseAsIs <0x000f4>
	  at System.Net.Sockets.Socket:Dispose <0x00426>
	  at System.Net.Sockets.Socket:Dispose <0x000a4>
	  at System.Net.Sockets.Socket:Close <0x00098>
	  at System.Net.Sockets.TcpClient:Dispose <0x0011c>
	  at System.Net.Sockets.TcpClient:Dispose <0x0001a>
	  at System.Net.Mail.SmtpConnection:ShutdownConnection <0x00184>
	  at System.Net.Mail.SmtpConnection:Abort <0x00012>
	  at System.Net.Mail.SmtpTransport:Abort <0x00078>
	  at System.Net.Mail.SmtpClient:Abort <0x0001c>
	  at System.Net.Mail.SmtpClient:SendAsyncCancel <0x00088>
	  at <>c:<SendMailAsync>b__84_1 <0x0001c>
	  at System.Threading.CancellationTokenSource:Invoke <0x00042>

or

#9  0x00007fbc510f8102 in __GI___assert_fail (assertion=0x7fbc4dcdc8a6 "0 <= fd && fd < sysconf(_SC_OPEN_MAX)", file=0x7fbc4dcddbb1 "/__w/1/s/src/native/libs/Common/pal_utilities.h", line=86, function=0x7fbc4dcdd527 "int ToFileDescriptor(intptr_t)") at assert.c:101
#10 0x00007fbc4dceb7cd in ToFileDescriptor (fd=-1) at /__w/1/s/src/native/libs/Common/pal_utilities.h:86
#11 0x00007fbc4dcebd8e in SystemNative_SetLingerOption (socket=-1, option=0x7fbc45d84d18) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:1278
...
	  at <unknown> <0xffffffff>
	  at Sys:<SetLingerOption>g____PInvoke|34_0 <0x00024>
	  at Sys:SetLingerOption <0x00068>
	  at System.Net.Sockets.SocketPal:SetLingerOption <0x00092>
	  at System.Net.Sockets.Socket:SetLingerOption <0x00022>
	  at System.Net.Sockets.Socket:SetSocketOption <0x00248>
	  at System.Net.Sockets.Socket:set_LingerState <0x00024>
	  at System.Net.Sockets.TcpClient:set_LingerState <0x00022>
	  at System.Net.Mail.SmtpConnection:ShutdownConnection <0x000f0>
	  at System.Net.Mail.SmtpConnection:Abort <0x00012>
	  at System.Net.Mail.SmtpTransport:Abort <0x00078>
	  at System.Net.Mail.SmtpClient:Abort <0x0001c>
	  at System.Net.Mail.SmtpClient:SendAsyncCancel <0x00088>
	  at <>c:<SendMailAsync>b__84_1 <0x0001c>
	  at System.Threading.CancellationTokenSource:Invoke <0x00042>

Note the fd is -1.

Are the "long running" cases where the assertion has popped a dialog or something? Or different issue?

@noahfalk
Copy link
Member

noahfalk commented Jul 26, 2022

This afternoon's rolling build just failed in the System.Net.Http.Functional.Tests test (http not mail), but the error in the log has a very similar pattern of 7 iterations of cancellation tests 2 minutes apart followed by a timeout.
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-287f10d639984af991/System.Net.Http.Functional.Tests/1/console.8747f084.log?%3Fhelixlogtype%3Dresult

   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:02:17
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:04:18
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:06:19
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:08:20
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:10:21
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:12:22
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException', Elapsed: 00:14:23
['System.Net.Http.Functional.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]

This morning it was Linux x64, now it is OSX x64.

@noahfalk noahfalk changed the title system.net.mail.functional.tests fail in CI System.Net.* cancelation tests fail in CI Jul 26, 2022
@noahfalk
Copy link
Member

@danmoseley - For the moment I am guessing we've potentially got multiple failures hitting the same test. I opened #72830 to track the Assertion `0 <= fd && fd < sysconf(_SC_OPEN_MAX) issue. If it turns out the cancellation and the fd have the same root cause we can always dupe it.

@noahfalk
Copy link
Member

@dotnet/ncl

@noahfalk
Copy link
Member

The failures in this issue also look similar to failures in System.IO.Pipes.Tests (#72585) which recently had tests disabled.

@noahfalk noahfalk changed the title System.Net.* cancelation tests fail in CI System.Net.* cancellation tests fail in CI Jul 26, 2022
@stephentoub
Copy link
Member

The failures in this issue also look similar to failures in System.IO.Pipes.Tests (#72585) which recently had tests disabled.

The HTTP failures are the same as #72596 (these are just the ValueTask versions of those same tests). @rzikm, can you please disable these variants as well against the original issue?

@CarnaViire
Copy link
Member

System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadAsync_CancelPendingValueTask_ThrowsCancellationException disabled in #72854

@noahfalk
Copy link
Member

System.Net.Mail.Functional tests are still failing in CI as of this morning. https://runfo.azurewebsites.net/view/build/?number=1909685.

@noahfalk
Copy link
Member

noahfalk commented Aug 3, 2022

System.Net.Mail.Functional tests are still failing in CI. I submitted PR #73340 to disable the test.

noahfalk added a commit to noahfalk/runtime that referenced this issue Aug 4, 2022
This test is failing jobs in CI, see issue dotnet#72818
@karelz karelz added the untriaged New issue has not been triaged by the area owner label Aug 4, 2022
@karelz
Copy link
Member

karelz commented Aug 4, 2022

Ah, @antonfirsov accidentally removed untriaged without setting milestone, so it did not show up in our Triage queries nor 7.0 work :(

@karelz
Copy link
Member

karelz commented Aug 4, 2022

@noahfalk we will need to somehow break down this mega issue - it is rather confusing.
I see that you didn't track failures in the OneNote for Test Monitor. Do you have other ways you keep list of frequency? (Runfo is sadly not helpful here :()

rzikm pushed a commit that referenced this issue Aug 4, 2022
This test is failing jobs in CI, see issue #72818
@noahfalk
Copy link
Member

noahfalk commented Aug 4, 2022

At the moment this issue is intended to track exactly the two failures listed in first comment, and I am hoping that with PR #73340 it is now mitigated. If any other tests fail with a similar issue I can open a new issue to track them. There was also the assert 0 <= fd thing which was a different error occuring in the same tests which has a separate issue already tracking it.

Do you have other ways you keep list of frequency?

I was relying on runfo's updating the GitHub issue with a list of builds where it occurred + looking at the most recent few rolling builds in runfo. I was also looking into potential runfo UI changes because I agree as the test monitor it isn't trivial to determine which issues remain hot at a glance.

@noahfalk
Copy link
Member

noahfalk commented Aug 4, 2022

I'm waiting to see the afternoon rolling build, but assuming we see no failures there I will remove the blocking CI tag.

@karelz
Copy link
Member

karelz commented Aug 5, 2022

Yeah, Runfo is really not useful for tracking these kind of failures (i.e. not just specific test) -- we did it by hand until now to avoid confusion.
For example, there is no clarity on frequency of each of the failures -- therefore it is hard to determine how much it impacts CI.
This kind of info keeps getting lost during Test Monitor transitions :( ... I will try to backfill some of it for last week to make next Test Monitor (@akoeplinger) life easier.

@karelz
Copy link
Member

karelz commented Aug 5, 2022

Closing this issue as it tracks basically anything wrong in Networking, which is not useful.
If there are specific issues still occurring, we should create issues with history of failures like the 2 above.

@karelz karelz closed this as completed Aug 5, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Aug 5, 2022
@karelz karelz added this to the 7.0.0 milestone Aug 7, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Sep 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Net blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Projects
None yet
Development

No branches or pull requests

6 participants