Test not run with IEnumerable tuple arguments from TestCaseSource

[denvermadsen]

Environment:

• NUnit v3.12.0
• NUnit3TestAdapter v3.17.0
• Microsoft.NET.Test.Sdk v16.7.1
• Visual Studio Community 2019 16.7.2
• .NET Core 3.1

Summary:
Unit tests parameterized with an IEnumerable of tuples (via TestCaseSource) is not run with Test Explorer's Run All button.

Expected Behavior:
All test cases are enumerated in Test Explorer with Fail or Pass statuses upon pressing Run All. In sample provided, it is expected that the test case will fail with a red X badge in Test Explorer.

Current Behavior:
Test Cases are not run, status for tests are unfilled blue badges. Outputs the following error:
========== Test discovery finished: 1 Tests found in 686 ms ========== ---------- Starting test run ---------- NUnit Adapter 3.17.0.0: Test execution started An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed. ========== Test run finished: 0 Tests run in 460.3 ms (0 Passed, 0 Failed, 0 Skipped) ==========

Additional Notes:
A workaround is observed: test runs if using TestCaseData.SetArgDisplayNames()

Repro steps:
Either: open solution in Sample.zip and run tests in Test Explorer
OR:

1. Create new .NET Core 3.1 console project
2. Add NUnit Test project to solution (w/ Nuget packages Microsoft.NET.Test.Sdk, NUnit and NUnit3TestAdapter )
3. Add unit test method with parameter of type IEnumerable<(object, object)> and method attribute [TestCaseSource(...)]
4. For target class of TestCaseSource, implement IEnumerable interface and return a test case with tuple, e.g. yield return new TestCaseData(new[] {(new object(), new object())});
5. Open Visual Studio Test Explorer and press Run All.
6. Observe blue unfilled status icons of tests, error in Output/Tests
   
   

[OsirisTerje]

@denvermadsen
Thanks for the issue and repro!

This is part of the issues we have with FQNs. I'll mark it as one more of those.

The workaround for now is to add a runsettings file with the following two options in the NUnit section:

      <UseParentFQNForParametrizedTests>true</UseParentFQNForParametrizedTests>
      <UseNUnitIdforTestCaseId>true</UseNUnitIdforTestCaseId>

[zhaparoff]

It seems I've faced the similar issue when using IEnumerable<record_type> as TestCaseSource input.

• NET 5
• VS Ent 2022 17.1.6

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.2.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
    <PackageReference Include="coverlet.collector" Version="3.1.2">

    public record SampleTestCase(int? A, int? B, int ExpectedResult, string Because = null);

    public class SampleTestCases
    {
        public static IEnumerable<SampleTestCase> GetTestCases(bool includeNullValuesForA)
        {
            if (includeNullValuesForA)
            {
                yield return new SampleTestCase(null, null, 0);
                yield return new SampleTestCase(null, 2, -1, "null value should have precedence over others");
            }

            yield return new SampleTestCase(1, null, 1, "null value should have precedence over others");
            yield return new SampleTestCase(1, 2, -1);
            yield return new SampleTestCase(4, 2, 1);
            yield return new SampleTestCase(4, 4, 0);
        }
    }

    [TestFixture]
    public class SampleComparerTests
    {
        [TestCaseSource(
            typeof(SampleTestCases),
            nameof(SampleTestCases.GetTestCases),
            new object[] { true })]
        public void SampleComparableTestA(SampleTestCase testCase)
        {
            testObj.Compare(testCase.A, testCase.B).Should().Be(testCase.ExpectedResult, testCase.Because);
        }

        [TestCaseSource(
            typeof(SampleTestCases),
            nameof(SampleTestCases.GetTestCases),
            new object[] { false })]
        public void SampleComparableTestB(SampleTestCase testCase)
        {
            testObj.Compare(testCase.A, testCase.B).Should().Be(testCase.ExpectedResult, testCase.Because);
        }
    }

In case when running all tests only first yield is running the test, others are skipped.
Moreover, when test case set is used in multiple test methods - it will run only the first one, others will be skipped entirely.
In the example above only one yield will be run for SampleComparableTestA and no test cases will be run for SampleComparableTestB at all.

In the test output window I can see that only 1 test is discovered by the execution engine, although Visual Studio UI shows all 6 cases available.

NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run

Meanwhile, all test cases are yielded, discovered and run in Live Unit Testing Mode, without issues.

Suggested workaround with .runsettings parameters for NUnit didn't work for me.