Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotnet test <dll> --filter parameter is ignored when there are a large number of tests. All tests in all classes are run regardless of filter. #1036

Open
fenjas opened this issue Dec 1, 2022 · 6 comments

Comments

@fenjas
Copy link

fenjas commented Dec 1, 2022

  • NUnit and NUnit3TestAdapter versions: 3.13.3 and 4.3.1
  • Visual Studio edition and full version number (see Help About): VS 2022 17.4.2 (v 4.8.04084)
  • What .net platform and version is being targeted: .NET 6.0

I have a VS C# solution with 3 test classes as per pic attached. I'm using a PowerShell script to execute these tests on a schedule using the following statements:

  • dotnet test biditests.dll --filter TestCategory~Daily
  • dotnet test biditests.dll --filter TestCategory~Monthly
  • dotnet test biditests.dll --filter TestCategory~GroupBrandNames

image

The --filter parameter seems to be ignored, so when I run the dotnet command targeting ~Daily, all the tests in all 3 classes are executed instead of the ones targeted by the filter.

If I execute the dotnet command targeting ~GroupBrandNames or ~Monthly, the filter works fine presumably because there are only 2 and 60 tests respectively in each class. If I keep the no. of tests in the Daily class under 2000, everything works as expected. As of now it contains in excess of 3000 tests.

I also tried other ways to run the tests using the project instead of the the dll, and changing the filter to using the FQDN, etc. However, the issue is the number of tests.

I found an article which mentioned creating a .runsettings file and upping the AssemblySelectLimit from 2000 to something greater than the number of tests. I did this, however it does not seem to work (assuming I did it correctly, file attached) and from what I understood applies only when you're running tests from the IDE which is not my case.

As a fix of sorts, I ended up using a Environment.Exit() call in the OneTimeTearDown method after the last test fixture is executed so the runner exits before getting a chance to run tests from another class. It's ugly but it works for now.

image
image
image

Logging this here since I found a similar issue and thought mine would be related!

Thanks.

@OsirisTerje
Copy link
Member

  1. Add a runsettings file with Verbosity=5 and see what the filter expression internally is.
  2. Try to use the NUnit Where syntax and see what that gives and compare the filter.
  3. See if you can provide a simple repro solution for this

Information about the settings is here: https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html

@fenjas
Copy link
Author

fenjas commented Dec 1, 2022

  1. ok, right now verbosity is set to 0. Setting it to 5.
  2. ok
  3. Will try to

Thanks!

@OsirisTerje
Copy link
Member

Just in case: To get the output from dotnet test, remember to set the consolelogger.
E.g.:
image

@fenjas
Copy link
Author

fenjas commented Dec 2, 2022

Turns out that adding -s .runsettings to the dotnet command with AssemblySelectLimit set to a value greater than the total number of tests does seem to address the issue. Regardless, I created a C# solution demonstrating this behaviour and 'fix' -> https://github.com/fenjas/Nunit2000TestsLimit

At the risk of being pedantic though, I noticed that the total number of Passed tests is the same in both cases i.e when the filter is applied and when it is ignored. See the output in the no_runsettings.txt and with_runSettings.txt included.

I still have to try this out on the actual tests I run live so will follow up once I do.

Test Run Successful.
Total tests: 2280
Passed: 2280
Total time: 2.6461 Seconds

Test Run Successful.
Total tests: 2280
Passed: 2280
Total time: 3.1467 Seconds

@torers-dnv
Copy link

Yes, seems as filter behavior changes when exceeding 2000 tests. Filter is applied on results after running tests rather than excluding tests from running in the first place, which is weird and unexpected.
Thank you for the AssemblySelectLimit workaround @fenjas , it reduced the build time by 12 minutes on our TeamCity build.

@OsirisTerje
Copy link
Member

If you're using dotnet test you can also set the limit directly without creating a .runsettings file.

dotnet test <somefilterexpression> -- NUnit.AssemblySelectLimit=99999

Also see https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#assemblyselectlimit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants