System.Net.NetworkInformation.Tests.PingTest.SendPingWithHostAndTimeoutAndBufferAndPingOptions and others fail on AppleTV

[karelz]

Occurrences 7/5-8/5 (incl. PRs) via Kusto and Runfo last 30 days:

• First failure on 7/27 - happening 3x-4x per week

Day	Run	OS
8/6	Rolling run	OSX.1100.Amd64.AppleTV.Open
8/4	PR #73362	OSX.1100.Amd64.AppleTV.Open
8/2	PR #73023	OSX.1100.Amd64.AppleTV.Open
8/1	PR #71203	OSX.1100.Amd64.AppleTV.Open
7/27	PR #70015	OSX.1100.Amd64.AppleTV.Open
7/26	Rolling run	OSX.1100.Amd64.AppleTV.Open

Test: System.Net.NetworkInformation.Tests.PingTest.SendPingWithHostAndTimeoutAndBufferAndPingOptions

Assert.Equal() Failure
Expected: Success
Actual:   TimedOut

   at System.Net.NetworkInformation.Tests.PingTest.PingResultValidator(PingReply pingReply, IPAddress[] localIpAddresses, ITestOutputHelper output)
   at System.Net.NetworkInformation.Tests.PingTest.<>c__DisplayClass30_0.<SendPingWithHostAndTimeoutAndBufferAndPingOptions>b__1(PingReply pingReply)
   at System.Net.NetworkInformation.Tests.PingTest.SendPing(Func`2 sendPing, Action`1 pingResultValidator)
   at System.Net.NetworkInformation.Tests.PingTest.SendBatchPing(Func`2 sendPing, Action`1 pingResultValidator)
   at System.Net.NetworkInformation.Tests.PingTest.SendPingWithHostAndTimeoutAndBufferAndPingOptions()
   at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr)

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | where Branch <> 'refs/pull/73374/merge'
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    '', //testNameSubstring
    'SendPingWithHostAndTimeoutAndBufferAndPingOptions', //methodName
    'TimedOut', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Occurrences 4/5-8/5 (incl. PRs) via Kusto and Runfo last 120 days:

• First failure on 7/27 - happening 3x-4x per week

Day	Run	OS
8/6	Rolling run	OSX.1100.Amd64.AppleTV.Open
8/4	PR #73362	OSX.1100.Amd64.AppleTV.Open
8/2	PR #73023	OSX.1100.Amd64.AppleTV.Open
8/1	PR #71203	OSX.1100.Amd64.AppleTV.Open
7/27	PR #70015	OSX.1100.Amd64.AppleTV.Open
7/26	Rolling run	OSX.1100.Amd64.AppleTV.Open

Test: System.Net.NetworkInformation.Tests.PingTest.SendPingWithHostAndTimeoutAndBufferAndPingOptions

Assert.Equal() Failure
Expected: Success
Actual:   TimedOut

   at System.Net.NetworkInformation.Tests.PingTest.PingResultValidator(PingReply pingReply, IPAddress[] localIpAddresses, ITestOutputHelper output)
   at System.Net.NetworkInformation.Tests.PingTest.<>c__DisplayClass30_0.<SendPingWithHostAndTimeoutAndBufferAndPingOptions>b__1(PingReply pingReply)
   at System.Net.NetworkInformation.Tests.PingTest.SendPing(Func`2 sendPing, Action`1 pingResultValidator)
   at System.Net.NetworkInformation.Tests.PingTest.SendBatchPing(Func`2 sendPing, Action`1 pingResultValidator)
   at System.Net.NetworkInformation.Tests.PingTest.SendPingWithHostAndTimeoutAndBufferAndPingOptions()
   at System.Reflection.MethodInvoker.InterpretedInvoke(Object obj, Span`1 args, BindingFlags invokeAttr)

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | where Branch <> 'refs/pull/73374/merge'
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    '', //testNameSubstring
    'SendPingWithHostAndTimeoutAndBufferAndPingOptions', //methodName
    'TimedOut', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

Author:	karelz
Assignees:	-
Labels:	area-System.Net, runtime-mono, os-tvos
Milestone:	-

[karelz]

@lewing can you please triage this one? Thanks!

[akoeplinger]

I expanded the Kusto query a bit and found the same failure in other PingTest tests on AppleTV:

SendPingAsyncWithHostAndTimeout
SendPingAsyncWithHostAndTimeoutAndBuffer
SendPingWithHostAndTimeoutAndBuffer
SendPingWithHostAndTimeoutAndBufferAndPingOptions
SendPingWithIPAddressAndTimeoutAndBufferAndPingOptions_Unix

The common thing seems to be using "localhost" to ping instead of an IP address. It also seems to happen on only a subset of the machines which suggests an infra issue:

DNCENGMAC109.local	4
DNCENGMAC101.local	4
DNCENGMAC092.local	3
DNCENGMAC100.local	2
DNCENGMAC016.local	1
DNCENGMAC104.local	1
DNCENGMAC024.local	1
DNCENGMAC059.local	1

[mdh1418]

Hit this on an unrelated PR on DNCENGMAC092.local #76465

[steveisok]

I'm not seeing any hits on this, so closing.